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ABSTRACT 



The Interdisciplinary Teamed Instruction (ITI) Project was a 
2 -year project aimed at determining the effects of ITI on teaching and 
learning and at validating the effectiveness of a professional development 
model to facilitate development, implementation, and evaluation of ITI. 
Through summer institutes and onsite workshops, project staff provided 
training, resources, and ongoing technical assistance to 

teacher/administrator teams from four rural Virginia secondary schools . Teams 
designed and tested integrated curricula that incorporated learning 
activities and alternative assessment forms; conducted collaborative action 
research to determine the effectiveness of units; and redesigned units using 
feedback from students, teachers, parents, and supervisors. Researchers 
evaluated the 1996 ITI Summer Institute. Evaluation included questionnaires 
on teacher empowerment, schools as learning communities, school 
effectiveness, and teachers' sense of control; observations; participant 
interviews, reflection, and feedback; oral evaluation feedback; and rating of 
participants' videotaped presentations. Results indicated that the Institute 
met its intended outcomes. The greatest impact on participants related to 
collaboration, cooperative learning, assessment rubrics, multiple 
intelligences, and team building. Participants were very positive in 
evaluating the Institute, which strengthened their belief in and commitment 
to ITI concepts and practices. The quantitative instrumentation utilized may 
not have fully captured and documented Institute effects. Appendices contain 
a meeting agenda; three questionnaires on teachers, school as community, 
school products and services; a control questionnaire; a feedback form; 
interview questions; a team reflection log; presentation scoring rubric; and 
an evaluation standards checklist. (Contains 20 references.) (SM) 
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EXECUTIVE SUMMARY 



The Interdisciplinary Teamed Instruction (ITI) project was a 2-year (1992-1994) research 
and development project aimed at determining the effects of interdisciplinary teamed instruction on 
teaching and learning. The study also sought to validate the effectiveness of a professional 
development model that would facilitate development, implementation, and evaluation of ITI. 
Through summer institutes and on-site workshops, project staff provided training, resources, and 
ongoing technical assistance to teams of teachers and administrators from four rural Virginia 
secondary schools. These teams designed and tested integrated curricula that incorporated learning 
activities and alternative forms of assessment; engaged in collaborative action research to determine 
the effectiveness of units; and redesigned units using data-based feedback from students, teachers, 
parents, and supervisors (Bums, 1994). Following the initial ITI development, Rebecca Bums, the 
ITI project director, wrote Dissolving the Boundaries: Planning for Curriculum Integration in 
Middle and Secondary Schools (1995). This book, with the Facilitator's Guide, is a professional 
development tool designed to increase a faculty's understanding of curriculum integration and to 
assist them with decision making about ITI implementation. 

To assist schools that were ready to design and implement ITI, three summer institutes 
were held between June 1995 and August 1996 to help teams of teachers and administrators from 
14 schools (representing five states) build capacity for collaboration and interdisciplinary teamed 
instruction. Two of the institutes were held in 1995: The first was intended to field test the ITI 
concept outside the parameters of a research and development project, and included six schools 
from three states. The second was a contracted activity conducted for one school in West 
Virginia. The third institute, held at Radford University in Virginia from July 7-12, 1996, is the 
basis for this report. The purpose of the Institute was to help participants design integrated units, 
plan learning activities and assessments, and present their units as a culminating activity at the 
close of the training. Thirty participants from four states attended this Institute. 

The major purpose of this effort was to conduct a comprehensive evaluation of the 1996 
ITI Summer Institute. This evaluation focused on the processes, outcomes, and products of the 
Institute. Key evaluation activities included administering questionnaires on teacher empower- 
ment, schools as communities of learners, school effectiveness, and teachers’ sense of control; 
on-site observations; face-to-face interviews with participants; reflective assignments for 
participants; participant feedback forms; oral evaluation feedback provided during the Institute, 
and review and rating of participants’ videotaped presentations. 

The major objectives of this evaluation included the following: to measure participants’ 
sense of personal locus of control; to evaluate the processes of the ITI Institute via participants’ 
reflective assignments, feedback forms, personal interviews, and observations of activities; to 
evaluate participants’ culminating activity of videotaped presentations, completed in teams; to 
assess the outcomes of the ITI Institute on participants sense of teacher empowerment and 
professional learning community; and to assess the outcomes of the ITI Institute on participants’ 
perceptions of their organization’s effectiveness. 

Databases were created using SPSS Windows for each of the quantitative instruments. 
After cleaning and verification, appropriate analyses were conducted, including internal 
consistency reliabilities, descriptive statistics, matched pair /-tests, and effect sizes. Qualitative 
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data were categorized and summarized, and reported accordingly. The videotaped presentations 
were reviewed and rated using a scoring rubric provided by Institute staff. 

Conclusions. The conclusions are presented by type of data collection, followed by 
overall conclusions. 

Teacher Questionnaire. We conclude that the 1 996 ITI Summer Institute had some effect 
on the Self-Efficacy, Decision Making, Impact, and Autonomy in Scheduling subscales after a 
period of 1 year. For the Total Scale, the Institute had some effect, and this effect was almost 
moderate after 1 year. The Institute had little effect on the Status and Professional Growth 
subscales after 1 year. 

School as Community Questionnaire. We conclude that the 1996 ITI Summer Institute 
had moderate effect on the Collaborative Activity subscale and some effect on the Total Score 
by the end of the Institute. The Institute had some effect on the Collaborative Activity and 
Collective Focus on Student Learning subscales after a period of 1 year. The Institute had little 
effect on the Shared Sense of Purpose and Reflective Dialogue subscales and the Total Score after 
1 year. The Institute had no effect on the Deprivatized Practice subscale after 1 year. 

School Products and Services Questionnaire. We conclude that the 1996 ITI Summer 
Institute had little effect on the Total Score after a period of 1 year. 

Control Questionnaire. We conclude that the 1996 ITI Summer Institute teachers scored 
higher on the Self-Confidence subscale in comparison to the Autonomous Behavior subscale. 
And, when the subscales were combined, the teachers scored moderately high on the Total Score. 

Feedback Form. We conclude that participants felt the 1 996 ITI Summer Institute greatly 
met its intended outcomes. They found the Institute to be well planned, informative, 
motivational/inspirational, and worth the time and money invested. The most meaningful 
Institute activities for participants were team building, assessment rubrics, and multiple 
intelligences; major strengths were team building, information provided, and the organization/ 
implementation. The Institute positively changed participants’ thinking, resulting in increased 
belief in and reinforcement of ITI principles, and heightened awareness of its potential uses. 
Furthermore, participants were eager to incorporate their new knowledge back in their respective 
schools via rubrics, multiple intelligences, cooperative learning, and team building. 

Role Alike Caucus Activity. We conclude that participants of the 1996 ITI Summer 
Institute felt the most promising new approaches to teaching and learning included technology 
and learning styles and strategies. These styles and strategies included higher-level thinking, 
questioning techniques, group learning, cooperative learning, and hands-on learning. We also 
conclude that collaborating to design interdisciplinary studies brings about opportunities to those 
involved including collaboration, networking, and sharing new resources and ideas with others. 
And, conversely, this collaboration invokes challenges such as time restrictions, scheduling 
problems, and persuading others to become risk takers. 

Interviews. We conclude that most participants of the 1996 ITI Summer Institute felt a 
great deal of freedom in deciding what and how curriculum is taught in their respective schools. 
Also, that participants advocated using strategies such as critical thinking and analysis skills, 
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and projects, cooperative learning, and computer training to help students become independent 
learners. The majority of Institute participants described student learning in their respective 
schools as positive or average, with individual achievement levels differing by student. Most 
teams felt that their level of collaboration was good, if not excellent. About half of the 
participants felt their respective school faculties were ready and willing to implement school 
change. Another fourth felt that about half their faculties were ready at this point, but the 
remaining fourth felt implementation would be difficult at the present time. 

Team Reflection Logs. We conclude that, as the 1 996 ITI Summer Institute progressed, 
participants moved from learning and gathering information to clarifying roles and collaborating 
to designing activities to reviewing and revising and, finally, to integrating the unit. As a result, 
their emotions ranged widely from curiosity, apprehension, and anticipation at the beginning to 
enthusiasm and understanding midway through to exhausted, relieved, and productive at the 
conclusion. The Institute served as a positive modeling experience for participants, and they felt 
a sense of accomplishment upon mastering assessment rubrics. Further, participants felt the most 
beneficial Institute activities were multiple intelligences, rubrics, the mousetrap, and jigsaw. 

Overall Conclusions. We conclude, from the triangulation of the data garnered from the 
mixed-method approach, that the 1996 ITI Summer Institute had the greatest impact on 
participants in the following areas: collaboration, cooperative learning, assessment rubrics, 
multiple intelligences, and team building. Participants were overwhelmingly positive in their 
valuation of the Institute, which strengthened their belief in and commitment to ITI concepts and 
practices. Finally, after reviewing the outcomes of this comprehensive evaluation, we conclude 
that the quantitative instrumentation utilized may not have fully captured and documented 
institute effects. 

Recommendations. A number of recommendations were offered, and are listed below. 
We recommend that the program continue offering ITI summer institutes to interested educators. 
Suggested improvements for future institutes might include more explanatory content, more time 
for individual activities, and slight scheduling adjustments. Also, we recommend that the 
program continue supporting curriculum integration efforts by keeping in touch with participants 
and providing new resources when available. One approach for this communication and 
information provision that is already being implemented by ITI staff is the use of their electronic 
ITI listserv, which connects interested educators from around the world. 

We recommend that the program continue evaluating such institutes, with both qualitative 
and quantitative methods. We do not feel the need, however, for such an intensive, compre- 
hensive evaluation effort as that presented in this report. The extensive evaluative methodology 
did not seem to be efficacious, in that the amount of information gained did not seem to be worth 
the prodigious amount of time spent gathering and analyzing data. Further, the utility of the three 
quantitative instruments used in this evaluation is questionable, since these surveys did not 
specifically assess the concepts and constructs undergirding the ITI activities and philosophies. 
We recommend that staff locate and secure other relevant instrumentation for future endeavors. 
And, staff might even consider alternate ways of focusing evaluative efforts to better document 
ITI’s ultimate impact on student outcomes. Staff might want to investigate in depth why some 
schools succeed in their ITI endeavors and report continuing implementation, whereas others 
apparently do not demonstrate similar success. One such approach might be site visits at selected 
ITI schools whose faculty have participated in previous summer institutes. 
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INTRODUCTION 



Interdisciplinary Teamed Instruction Background 

The Interdisciplinary Teamed Instruction (ITI) project was a 2-year ( 1 992- 1 994) research 
and development project aimed at determining the effects of interdisciplinary teamed instruction 
on teaching and learning. The study also sought to validate the effectiveness of a professional 
development model that would facilitate development, implementation, and evaluation of ITI. 
Through summer institutes and on-site workshops, project staff provided training, resources, and 
ongoing technical assistance to teams of teachers and administrators from four rural Virginia 
secondary schools. These teams designed and tested integrated curricula that incorporated 
learning activities and alternative forms of assessment; engaged in collaborative action research 
to determine the effectiveness of units; and redesigned units using data-based feedback from 
students, teachers, parents, and supervisors (Bums, 1994). 

Program participants reported positive effects of ITI on teaching at the four sites, 
particularly in terms of professional growth, reflective practice, and collegial interaction. They 
also reported positive effects of ITI on student attitudes and performance, particularly for lower- 
achieving students; rated the project's professional development model of summer institutes, on- 
site workshops, and action research as highly effective; and described conditions that enhanced 
successful ITI implementation (Bums, 1994). 

Participants made two major recommendations based on these conditions: (1) Before ITI 
is implemented, school faculty and community should (a) learn about ITI concepts, processes, and 
conditions; and (b) determine their readiness and desire to begin ITI. (2) AEL should develop 
resources to support ITI implementation (Bums, 1994). 

In response to these recommendations, Rebecca Bums, the ITI project director, wrote 
Dissolving the Boundaries: Planning for Curriculum Integration in Middle and Secondary 
Schools (1995). This book, with the Facilitator's Guide, is a professional development tool 
designed to increase a faculty's understanding of curriculum integration and to assist them with 
decision making about ITI implementation. Since the spring of 1995, AEL staff have conducted 
48 Dissolving the Boundaries professional development sessions in 12 states, as well as the 
District of Columbia and Puerto Rico. Additionally, the book is used by schools, districts, and 
colleges in 42 states, Puerto Rico, Nepal, and Argentina. 

To assist schools that were ready to design and implement ITI, three summer institutes 
were held between June 1995 and August 1996 to help teams of teachers and administrators from 
1 4 schools (representing five states) build capacity for collaboration and interdisciplinary teamed 
instruction. Two of the institutes were held in 1995: The first was intended to field test the ITI 
concept outside the parameters of a research and development project, and included four schools 
from Virginia, one from Tennessee, and one from Texas. The second was a contracted activity 
conducted for one school in West Virginia. The third institute, held in Virginia in July 1996, is 
the basis for this report. 
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1996 ITI Summer Institute 

The third ITI Summer Institute was held at Radford University in Virginia from July 7-12, 
1996. Facilitators included AEL staff member Rebecca Burns and AEL consultants Douglas 
Fleming and Barbara Fleming. Also in attendance was the AEL evaluator, Merrill Meehan, and 
the AEL scaling-up project director, Robert Childers. The purpose of the Institute was to help 
participants design integrated units, plan learning activities and assessments, and present their 
units as a culminating activity at the close of the training. 

Planning for the 1996 ITI Summer Institute focused on recruiting participants from sites 
where ITI had been introduced during the past year. Also, there was an emphasis on recruiting 
interns who might serve as trainers for future ITI institutes and workshops and who would 
promote the use of ITI in their locations. Recruitment was accomplished through personal contact 
by the ITI project leader, by telephone calls, and by mailing a descriptive brochure to other 
potential attendees (those who had sent teams to one of the 1995 institutes, who had purchased 
a copy of Dissolving the Boundaries , or who had requested information on ITI). One school 
district where Bums had conducted ITI workshops and one school that had participated in one of 
the 1995 ITI institutes enrolled teams; another site enrolled two interns. When AEL Board 
member Gary Ellerman of Radford University (VA) received information, he invited Burns to 
hold the Institute on the Radford campus. Subsequently, a Radford faculty member was accepted 
as an intern. 

Thirty participants from four states attended the 1996 ITI Summer Institute. Four high 
school teams from Mercer County, West Virginia attended. These teams (a total of 22 
participants) were part of a group that had participated in an ITI workshop in the spring of 1 996 
and would continue working with ITI through the 1996-97 school year. Four academic teachers 
and one vocational teacher from the county’s technical education center comprised each team. 
Two of the teams also included a building administrator. A fifth team, from Athens, Tennessee, 
was composed of four eighth-grade teachers and their principal. This was the second year that 
Athens Junior High had sent a team to an ITI institute. In 1995, the Athens principal attended 
with his seventh-grade team. Rounding out the Institute membership were three interns. One 
intern was a Radford University education department faculty member. The other two interns 
were curriculum specialists from the Oklahoma Department of Education who had participated 
in a May 1995 ITI workshop for their staff in Oklahoma City. 



Purpose of this Evaluation 

The major purpose of this effort was to conduct a comprehensive evaluation of the 1996 
ITI Summer Institute. This evaluation focused on the processes, outcomes, and products of the 
Institute. Key evaluation activities included administering questionnaires on teacher empower- 
ment, schools as communities of learners, school effectiveness, and teachers’ sense of control; 
on-site observations; face-to-face interviews with participants; reflective assignments for 
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participants; participant feedback forms; oral evaluation feedback provided during the Institute; 
and review and rating of participants’ videotaped presentations. This comprehensive evaluation 
of the 1996 ITI Summer Institute was viewed as a means of collecting baseline evaluative 
information from the first of a series of similar summer institutes planned for this contract period 
at AEL. Decisions regarding the evaluations of future ITI summer institutes were expected to be 
informed by the methodology and instrumentation employed in this evaluation. 



Major Objectives 

The major objectives of this evaluation included the following: 

• to measure participants’ sense of personal locus of control 

• to evaluate the processes of the ITI Institute via participants’ reflective assignments, 
feedback forms, personal interviews, and observations of activities 

• to evaluate participants’ culminating activity of videotaped presentations, completed 
in teams 

• to assess the outcomes of the ITI Institute on participants’ sense of teacher 
empowerment and professional learning community 

• to assess the outcomes of the ITI Institute on participants’ perceptions of their 
organization’s effectiveness 



Audience for this Report 

The primary audience for this report includes AEL staff and consultants, to provide input 
for modifying future institutes, and the Office of Educational Research and Improvement, AEL’s 
funding agency. Secondary audiences may include those interested in interdisciplinary teamed 
instruction, those interested in the effectiveness of summer institutes for teachers, and the actual 
participants of the 1996 ITI Summer Institute and their administrators. 
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METHODOLOGY 



This section presents the methodology used in completing the comprehensive evaluation 
of the ITI 1996 Summer Institute. Descriptions of the schedule, the data collection techniques, 
and the evaluation procedures are presented below. 



Institute Schedule 

The Institute was held at Radford University in Virginia, from July 7-12 (see Appendix A 
for a copy of the agenda). Registration and check-in began on Sunday afternoon, July 7, followed 
by an overview session covering introductions, expectations, team roles, team needs, networking, 
and journaling. Each participant received a copy of Burns’ book, Dissolving the Boundaries: 
Planning for Curriculum Integration in Middle and Secondary Schools. Participants also 
received notebooks containing additional handouts. 

Monday, July 8, began with an administrators’ breakfast, and then participants were asked 
to complete the “Teacher Questionnaire” and the “School as Community Questionnaire.” This 
was followed by a session on “Developing Teams: Roles, Rules, Relationships, Results.” After 
a mid-morning break, concurrent sessions were held on “Conflict Resolution,” “Tools for Testing 
Consensus,” and “Group Roles and Functions.” 

Following lunch, a “Team Exploration and Debriefing” session was conducted throughout 
the afternoon. This was a unique team-building session, in that each team was to design and 
construct a mousetrap-powered vehicle; these vehicles were then used in videotaped comparative 
trials with other teams. Mousetrap vehicles were constructed from materials such as cardboard, 
styrofoam, wood, plastic, rubber bands, string, cans, etc., and, of course, a mousetrap. In addition 
to team building, the session also introduced participants to the concept of performance 
assessment using a scoring rubric. Each vehicle was scored as either a Master, Journeyman, or 
Apprentice on five areas: (1 ) conforms with design specifications, (2) applies principles of force 
and motion, (3) shows testing and refinement using gathered data, (4) travels in a straight line, 
and (5) demonstrates reliability in use. Another “Team Planning and Reflections” session was 
held after dinner. 

An administrators’ breakfast was again held on Tuesday, July 9, followed by “Five Stages 
of Integration: Reviewing and Rating ITI Examples.” The second morning session was “Discov- 
ering ‘Fertile’ Themes.” After lunch, a “Team Applications: Unit Design” session was held, 
including “Generating Learner Goals” and “Identifying Powerful Learning.” The second 
afternoon session provided team planning time, with topics such as “Beginning the Unit 
Development — Selecting a Theme, Framing Essential Questions, and Designing Powerful 
Learning Experiences.” 
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Following the administrators’ breakfast on Wednesday, July 1 0, participants were asked 
to complete a “Control Questionnaire” and a “School Products and Services Questionnaire.” A 
short session was held afterward on “Learning Principles and Authentic Tasks.” Afterwards, 
concurrent sessions were held on “Cooperative Learning,” “Problem-Based Learning,” and 
“Multiple Intelligences.” Following the morning break, additional concurrent sessions were held 
on “Cooperative Tasks,” “Community-based Learning,” and “Writing to Learn.” The first 
afternoon session focused on “Structuring a Powerful Learning Experience,” followed by “Team 
Planning: Incorporating Authentic Tasks into Your Unit” and “Team Reflections.” 

Thursday, July 11, again began with an administrators’ breakfast, followed by two 
morning sessions: “Understanding Alternative Assessment” and “Constructing Scoring Rubrics.” 
After lunch, the entire afternoon session was devoted to “Team Planning Time: Construct a 
Rubric to Assess Student Performance, Construct a Rubric to Assess Unit Performance, and Team 
Reflection.” After dinner, teams worked on preparing their team exhibit. 



The last day of the Institute was Friday, July 1 2, which began with a final administrators’ 
breakfast. Two short morning sessions were held on “Planning for Re-Entry” and “Team Action 
Plans.” After the mid-morning break, team exhibitions were conducted and videotaped. After 
all teams were finished, certificates were presented, and then participants were asked to complete 
a “Feedback Form,” the “Teacher Questionnaire,” and the “School as Community Questionnaire.” 



Data Collection Techniques 

Data collection included a mix of quantitative and qualitative techniques, including four 
surveys, a feedback form, interviews, observations, videotaped presentations, and reflective 
activities. Each method is discussed below. 

Surveys. Four paper and pencil surveys were administered to participants in the 1 996 ITI 
Summer Institute. Each survey is discussed in turn. 

Teacher Questionnaire. The first survey was labeled the “Teacher Questionnaire” (see 
Appendix B for a copy of the questionnaire). This is AEL’s adaptation of the School Participant 
Empowerment Scale (SPES) originally developed by Short and Rinehart (1992). The SPES was 
administered to 4,091 teachers in 183 restructuring schools in Ohio by Klecker and Loadman 
(1 996), who re-factor analyzed the 38 items of the SPES into six subscales. All 38 items use the 
5-point Likert-type response options of 1 = Strongly Disagree to 5 = Strongly Agree. The six 
subscales and their Alpha internal consistency reliabilities were (1) Status, .80; (2) Professional 
Growth, .70; (3) Self-Efficacy, .89; (4) Decision Making, .80; (5) Impact, .83; and (6) Autonomy 
in Scheduling, .83. The number of items per subscale varied from a low of 3 for Autonomy in 
Scheduling to a high of 12 for Self-Efficacy. Klecker and Loadman (1998) conducted another 
factor analysis of the SPES using a subsample of 3,1 13 of the teachers in their larger sample. 
This second validity study confirmed their first study’s results. 
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AEL staff typeset the SPES items to fit on both sides of one 8 Vi" by 1 1" page using two 
columns per side. Through an error in printing, one item was included twice and another item 
was omitted. However, inspection of the duplicated item in Klecker and Loadman’s published 
article (1998, pp. 950-951) revealed that it actually loaded on all six factors. So, the conse- 
quences of counting it on two subscales were minimal. In addition to the 38 items, AEL staff 
added eight constructed-response demographic items at the end. These demographic questions 
asked respondents to supply their grades taught, subjects taught, years taught in any school, years 
taught in present school, employed full or part time, degree and credits earned, gender, and age. 

Table 1 displays the Cronbach Alpha internal consistency reliabilities for all four surveys 
employed in this evaluation. The Teacher Questionnaire Alphas are displayed first in Table 1, for 
each of its three administrations. Generally, the Alphas are acceptable, but some are higher than 
others. As expected, the full 38-item Alphas are high at .85 and .93, as are the 12-item Self- 
Efficacy subscale Alphas at .88 and .93. The 6-item Status subscale Alphas are .83, .89, and .78. 
Interestingly, the 3-item Autonomous Scheduling subscale has Alphas of .85, .90, and .80. The 
lowest Alphas are .59, .60, and .77 for the Professional Growth subscale. 

School as Community Questionnaire. The second survey was labeled the “School as 
Community Questionnaire” (see Appendix C for a copy of the questionnaire). This survey is 
AEL’s adaptation of school-wide professional learning community items included in a paper 
presented by Karen Seashore Louis and Helen Marks at the 1 996 American Educational Research 
Association (AERA) Annual Meeting in New York City. The construct of school-wide 
professional community is rather new in educational research. Louis and Marks’ work in the area 
grew out of connections with Fred Newmann and associates in a national R&D study of 
restructuring schools (Newmann & Wehlage, 1995). At the same time, Shirley Hord was 
conducting her research in the same general area, although she chose to label her construct Profes- 
sional Learning Community (1997). Hord developed an instrument to measure professional 
learning community that proved reliable and valid in a field test (Meehan, Orletsky, and Sattes, 
1997). Also, Royal, DeAngelis, and Rossi (1996) used items from the large, nationally 
representative Schools and Staffing Survey (SASS) from both 1 987-88 and 1993-94 to construct 
a measure of teachers’ sense of community within schools. 

AEL staff constructed the school-wide professional learning community instrument by 
adapting the 22 items that appeared in Louis and Marks’ 1996 American Educational Research 
Association paper and also in a subsequent article published in the American Educational Research 
Journal (Louis, Marks, & Kruse, 1996). In AEL’s adaptation of items, references to “Since the 
beginning of the school year . . .” were dropped and the response options were placed directly under 
the stem statement. The response option for the stem statements was a 5-point Likert-type scale from 
1 = Strongly Disagree (SD) to 5 = Strongly Agree (SA). The five subscales in the instrument are 
Shared Sense of Purpose, Collaborative Activity, Collective Focus on Student Learning, Deprivatized 
Practice, and Reflective Dialogue. The number of items per subscale varied from a low of 3 items 
(two scales) to a high of 6 items (two scales). The 22 items fit on both sides of one 8 Vi" x 11" page 
using two columns per side. As with the other instruments and because of the independent 
development of the measures, the remaining space was used to collect demographic information. 
These eight questions were the same as those included on the SPES instrument. 
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Table 1 : 

Alpha Reliabilities for the Four Instruments 
and Their Subscales by Administration Time 



Instrument Name 
Subscale Name 




Alpha Reliabilities 


In urn Dei 
of Items 


Pretest 

Administration 


End of 
Institute 


Posttest 

Administration 


Teacher Questionnaire 


38 


.93 


.93 


.85 


Status 


6 


.83 


.89 


.78 


Professional Growth 


4 


.59 


.60 


.77 


Self-Efficacy 


12 


.88 


.88 


.93 


Decision Making 


8 


.78 


.82 


.76 


Impact 


5 


.76 


.82 


.68 


Autonomy in Scheduling 


3 


.85 


.90 


.80 


School as Community Questionnaire 


22 


.92 


.93 


.90 


Shared Sense of Purpose 


3 


.57 


.78 


.86 


Collaborative Activity 


6 


.87 


.79 


.84 


Collective Focus on Student Learning 


4 


.60 


.70 


.75 


Deprivatized Practice 


3 


.65 


.66 


.69 


Reflective Dialogue 


6 


.89 


.87 


.62 


School Products and Services 
Questionnaire 


8 


.85 


N/A 


.91 


Control Questionnaire 


28 


.80 


N/A 


N/A 


Self-Confidence 


14 


.78 


N/A 


N/A 


Autonomous Behavior 


14 


.63 


N/A 


N/A 



0 
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Table 1 displays the Cronbach Alpha reliabilities for the total School as Community 
Questionnaire and its five subscales across three administrations. Generally, the Alphas are 
satisfactory, but some are higher than others. For example, the full 22-item Alphas were .92, .93, 
and .90 for the three administrations, respectively. The 6-item Collaborative Activity subscale 
Alphas are .87, .79, and .84, followed by those for Reflective Dialogue at .89, .87, and .62. The 
4-item Collective Focus on Student Learning subscale Alphas are .60, .70, and .75. As expected, 
the Alphas for the two 3-item subscales are a little lower, with only the pretest Alpha for the 
Shared Sense of Purpose subscale dipping into the .50s (.57). 

School Products and Services Questionnaire. The third survey was labeled the “School 
Products and Services Questionnaire” (see Appendix D for a copy of the questionnaire). This is 
AEL’s adaptation of the “Index of Perceived Organizational Effectiveness” (IPOE) instrument 
originally developed and published by Paul E. Mott (1972) and then modified for use in schools 
by Miskel, Fevurly, and Stewart (1979) and more recently by Hoy and Ferguson (1985). The 
IPOE is an eight-item, self-report instrument in which teachers rate the overall effectiveness of 
their school on four dimensions (quality and quantity of product, efficiency, adaptability, and 
flexibility) with two items for each dimension. Respondents select one of five options on a Low 
(1) to High (5) scale, but the response options/descriptors vary by the item stem. For example, 
for the first item on quantity of production, the responses range from “Low Production” to “High 
Production,” with three other descriptors between them. Possible scores range from a low of 8 
points to a maximum of 40 points; the higher the score on the IPOE, the greater the perceived 
organizational effectiveness of the school. 

AEL staff typeset the IPOE items to fit on both sides of one 8 Vi" x 1 1" page, five items 
on the front (preceded by instructions) and the remaining three on the back. Additionally, staff 
added six demographic questions on the back. Many of these were the same as those on the 
SPES, but role group was a new item. (The duplication of demographic items is a function of 
producing the instruments at different times.) 

The IPOE has been used in many studies over the years. Early studies reported high 
reliablity coefficients; for example, .89 by Miskel, Fevurly, and Stewart (1979). Similar 
reliability figures were reported by Hoy and Fergurson (1985) and, more recently, Loup and Ellett 
(1995) reported it to be .85. Table 1 shows that the School Products and Services Questionaire 
(IPOE) Alpha reliabilities in this study were .85 at pretest and .91 at post test. 

Control Questionnaire. The fourth survey was labeled the “Control Questionnaire” (see 
Appendix E for a copy of the questionnaire). This survey is AEL’s formatting of the “Internal Control 
Index” instrument developed by Pat Duttweiler (1984). The construct of locus of control is a 
personality trait that appears to influence human behavior across a wide variety of situations related 
to learning and performance. Much research on the locus of control construct has been completed 
over the past 30 years (Lefcourt, 1976; Rotter, Chance, & Phares, 1972). Locus of control relates to 
an individual’s expectancy for positive reinforcement. An individual with an internal control 
orientation believes that positive reinforcement depends on his or her own behavior whereas an 
individual with an external control orientation believes that positive reinforcement depends on chance, 
luck, or others (Rotter, Chance, & Phares, 1972). 
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The Internal Control Index was developed by Duttweiler (1984) to measure the internal 
locus of control orientation of adults. Duttweiler sought to improve on previous locus of control 
instruments that suffered from a variety of problems including low item-total score correlations, 
their forced choice format, response set, and the heterogeneity of the external control orientation. 
Duttweiler completed four distinct phases in the development of the new instrument: pretesting 
item development, tryout (pilot) testing, field testing, and a replication study. The final 
instrument consists of 28 different statements with one key word missing. The response format 
requires the respondent to choose one of five alternative words to insert into the statement. The 
five alternative words and their associated letters to insert in the blank space are Rarely (A), 
Occasionally (B), Sometimes (C), Frequently (D), and Usually (E). For example, the first item 

in the instrument is: “When faced with a problem I try to forget it.” The respondent chooses 

one of the five alternative words that best completes the statement and writes the letter for that 
word in the blank space. All 28 items fit on one page, printed front and back and, per 
Duttweiler’ s goal to eliminate response set, 14 of the items are reverse scored. AEL did not add 
demographic items to this particular instrument. 

Duttweiler (1 984) completed several reliability and validity studies on the Internal Control 
Index. The internal consistency reliability for the field-test phase (N = 548) was .84 and, for the 
replication study (N— 133), it was .85. As part of the validity checks, two factor analyses were 
completed. The first yielded two main subscales of Self-Confidence and Autonomous Behavior, 
while the second produced very similar results. The Self-Confidence subscale appears to measure 
confidence in one’ s own ability and it accounted for 69% of the common variance. This subscale 
consists of 14 of the instrument’s 28 items. The second subscale measures internally high 
responses to 14 other items that appear to represent a theme of Autonomous Behavior. That is, 
these 1 4 items portray behaviors that are “relatively independent of social reinforcement or social 
pressure” (p. 2 1 6). These 1 4 items accounted for the other 3 1 % of the common variance. As part 
of the convergent validity study, a significant negative correlation was found between the Internal 
Control Index and Mirel’s Factor I of the Rotter I-E Scale. Since the Rotter scale is a measure 
of external locus of control, the negative correlation with the Internal Control Index was expected. 

Table 1 displays the number of items and internal consistency reliabilities (Alphas) for 
the Self-Confidence and Autonomous Behavior subscales and total instrument as used in this 
evaluation (single administration only). Data in Table 1 show the 14-item Self-Confidence 
subscale Alpha is .78, the 14-item Autonomous Behavior subscale Alpha is .63, and the Alpha 
for the 28-item total instrument is .80. 

Feedback form. Participants were asked to complete this three-page form at the 
conclusion of the Institute (see Appendix F for a copy of the form). The first page contained 3 
demographic items (name, date, and position), followed by 15 closed-ended items, which 
participants were to complete by using a scale of 1 (Not at all) to 5 (Very much). The next two 
pages contained 4 open-ended items each, and provided ample space for participants’ responses. 
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Interviews and observations. The AEL evaluator conducted on-site interviews with 
participants (see Appendix G for a list of questions). These structured interviews took place 
during breaks, lunches, or after sessions were completed each day . Each interview consisted of 
five open-ended questions; responses were recorded on summary sheets. The AEL evaluator also 
attended most of the activities and observed both the proceedings and the participants reactions. 
Written notes were generated from this activity, and the evaluator met on-site with the presenters 
during the Institute (days and evenings) to provide formative evaluation information. 



Reflective activities. These activities included the “Role-Alike Caucus (see Appendix 
H for the activity sheet) and “Team Reflection Log” (see Appendix I for a copy of the log). For 
the role-alike activity, participants were asked to assemble into one of six role groups 
(administration, social studies, English/language arts, mathematics, science, and other). 
Participants were then given a set amount of time to respond as a group to two reflective 
questions. For the team reflection activity, individual teams were asked to meet briefly following 
each day’s activities (except for the last day) to debrief and think about their needs and concerns 
and then record their thoughts. 

Videotaped presentations. As the culminating activity, each team was asked to make 
a curriculum unit presentation at the conclusion of the Institute, and each presentation was 
videotaped. Criteria for assessing this activity were distributed and discussed with participants 
during the week. The evaluators adopted the Institute’s scoring rubric for their assessment of the 
teams’ presentations (see Appendix J for a copy of the rubric). 



Evaluation Procedures 

This section describes how data collection took place for each of the activities described 
earlier, and how resulting data were analyzed. 

Surveys. Each of the four surveys (Teacher Questionnaire, School as Community 
Questionnaire, School Products and Services Questionnaire, and Control Questionnaire) were 
administered on either Monday or Wednesday of the Institute. Participants were given time 
during the morning sessions to complete the instruments, which the AEL evaluator distributed. 
On Friday, the Teacher Questionnaire and School as Community Questionnaire were re- 
administered to participants during the morning session. Each survey was administered to all 
those participants present at each administration. 

The Teacher Questionnaire, School as Community Questionnaire, and School Products 
and Services Questionnaire were re-administered again in April 1997 to assess participants’ 
changes (growth or decline) in relevant areas. These surveys were mailed to respondents on April 
9 (with a cover letter and a self-addressed envelope), and a follow-up notice was sent in May. 
Of the 30 surveys mailed to participants, 19 usable questionnaires were returned for a return rate 
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of 63%. However, not every returned survey had complete data and, thus, the number of matched 
pairs for the t tests of subscales and total instrument scores varied from 14 to 18. 

Using SPSS, a statistical software program, the AEL evaluator created databases for each 
of the above surveys. Data were entered by AEL support staff, and were then cleaned and verified 
before analyses were conducted. The first step was to compute the internal consistency 
reliabilities (Cronbach Alpha) for each scale and subscale, where appropriate. Next, descriptive 
statistics were computed for each scale and subscale. This was especially important for the 
Control Questionnaire as it was administered only once. The t tests by the matched pairs method 
were computed for each applicable pair of scale and subscale scores. For the School Products and 
Services Questionnaire and the School as Community Questionnaire, two t tests were completed 
for each subscale and total scale score (pretest to end of Institute and pretest to posttest). The 
alpha level was set at .10. Last, effect sizes were computed for each t test and Cohen’s (1977) 
scheme for effect size descriptors was adopted. 

Feedback form. The Feedback Form was administered by the AEL evaluator to 29 
participants on Friday morning at the conclusion of the Institute. Again using SPSS, a database 
was created and data were entered. After cleaning, descriptive analyses were conducted for the 
15 Likert-type items on the first page of the survey. For the next two pages of open-ended 
responses, another AEL evaluator read the responses and assigned them to appropriate categories 
or themes. 

Interviews and observations. The on-site interviews were conducted by the AEL 
evaluator during the Institute. A summary sheet was completed for each of the 29 interviews, and 
these sheets formed the data for subsequent analysis. As well, notes taken for the evaluator’s 
observations during the Institute were reviewed and included in subsequent feedback sessions 
with the trainers as formative evaluation. These oral formative evaluation data are not included 
in this report. 

Reflective activities. The reflective activities were completed by participants during the 
Institute. Unlike the previous evaluation instruments, which were designed and administered by 
the evaluator, the reflective activities were designed and administered by the Institute staff. Then, 
the completed reflective activity outputs were collected and turned over to the evaluation staff as 
input to this evaluation. The role-alike activity took place early during the Institute, and 
completed group forms were submitted to the presenters. These summary sheets served as data 
for this analysis. Five team reflection logs from Sunday through Thursday were also gathered 
before the close of the Institute for subsequent analysis. 

Videotaped presentations. The participants were asked to make team exhibitions, which 
were videotaped by Institute staff. These videos were reviewed and rated by the AEL evaluators 
using a scoring rubric provided by Institute staff and shared with participants earlier. 
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FINDINGS 



This section presents the findings for each of the data collection techniques used in this 
comprehensive evaluation of the ITI 1996 Summer Institute, preceded by a discussion of the 
characteristics of the participants, gleaned from the demographic questions on three of the four 
surveys. 



ITI Institute Participants 

As stated above, 30 participants from four states attended the 1996 ITI Summer Institute. 
The majority of the participants (22) were from one county in West Virginia. They represented 
four high school teams comprised of four academic teachers and one vocational teacher from the 
county’s technical education center. Two of these West Virginia teams also included a building 
administrator. A fifth team was from a junior high school in Tennessee and included four eighth 
grade teachers and the principal. Three ITI interns participated in the 1 996 Summer Institute, one 
a faculty member from Radford University (Virginia) and two were curriculum specialists from 
the Oklahoma Department of Education. 

Of the 30 institute participants, 24 were female and 6 were male. Their ages ranged from 
24 to 62 years with a mode of 46.00, a mean of 48.43, and a median of 48.50. In terms of their 
educational level, one third possessed a master’s degree plus 45 credits, 7 had a master’s plus 30 
credits, 4 had a master’s plus 60 credits, and 3 had a master’s degree. One participant each 
reported possessing a bachelor’ s degree, a bachelor’ s plus 1 5 credits, a bachelor ’ s plus 45 credits, 
an educational specialist certificate, and a master’s plus 15 credits. All 30 participants reported 
working full time in their jobs. With respect to subjects taught, 5 indicated vocational subjects; 
4 each indicated math and/or computer, English/language arts, and history/social studies; 3 each 
indicated combination subjects, other (art, music, etc.), and administration; 2 indicated foreign 
languages; and 1 participant indicated science. One additional participant did not respond to this 
item. 



Participants reported teaching a total of 14 different grade levels and/or combinations of 
grade levels. The 9-12 grade-level group had the largest number of participants (11). The 10-12 
and 11-12 grade-level groups each had 3 participants in them, while all the others had just 1 or 
2 participants. On balance, the group was very experienced at teaching, with a mean of 22.00 
years and a maximum of 38.00 years. The minimum years of total teaching experience was 5.00, 
but the median was 22.50 and the mode was 19.00. The mean number of years of teaching in 
their current school was 8.57, with a minimum of 1.00 and a maximum of 22.00. The median 
years of teaching at their current school was 6.00 and the mode was 2.00. 
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Teacher Questionnaire 

Tables 2 and 3 display the results of the three administrations (pretest, end-of-institute, 
and posttest) of the Teacher Questionnaire to the ITI Institute participants. The six subscale 
results are presented in Table 2, while the Teacher Questionnaire Total Score results are presented 
in Table 3. The number of participants with both scores for the t tests varied from 16 to 28 for 
the subscales and was 1 4 and 23 for the Total Score. 

The Status subscale results are displayed first in Table 2* The possible score could range 
from 6 to 30 points. For the 27 participants with both pretest and end-of-institute scores, their 
pretest mean was 23.63 with a standard deviation of 3.61 . At the end-of-institute administration, 
their mean score declined 0.89 to 22.74, while the standard deviation increased to 4.13. The 
resultant t value was 2.22, which was significant at the .05 level. This is one of only two 
significant t values in Tables 2 and 3. The effect size was -0.247, which was “small” in Cohen’s 
(1977) terms. The pretest to posttest score also showed a small decline (0.28) from 23.72 to 
23.44. The standard deviation also declined slightly to 3.89. The / value was a very small 0.28, 
nowhere close to being significant. The effect size of -0.066 was much “less than small.” 

The Professional Growth subscale results are displayed second in Table 2. With the 
possible score ranging from 4 to 20 points, the pretest mean declined 0.14 at the end-of-institute 
administration. Also, the standard deviation declined slightly, as did the standard error of the 
mean. The resulting t value was very small at 0.45, which was not close to being significant. At 
-0.072, the effect size for this first t test for the Professional Growth subscale was much “less than 
small.” The pretest to posttest score increased slightly (0.22) to 17.56. The standard deviation 
and the standard error of the mean also declined. However, the 0.37 t value was not close to 
being significant. The 0.107 effect size was “less than small.” 

Self-Efficacy subscale results are displayed next in Table 2. The possible score could 
range from 12 to 60 points. The mean score in the first t test increased 0.23 to 49.69, w'hile the 
standard deviation decreased to 4.31. The resulting t value of just 0.36 was nowhere close to 
significance. The effect size of 0.047 was much “less than small.” The mean score in the pretest 
to posttest t test increased 1 .38 to 50.13, while the standard deviation decreased almost 1 point. 
At 0.97, the t value was not significant, but the 0.253 effect size was “small” in Cohen’s terms. 

The Decision Making subscale results are the fourth set of rows in Table 2. The possible 
score could range from 8 to 40 points. In the first t test, the mean score increased 1 .38 to 26.08, 
while both the standard deviation and the standard error of the mean decreased slightly. The t value 
was 2.15, significant at the .05 level. This is the second significant t value in Tables 2 and 3. At 
0.250, the effect size was “small.” The second t test increased by 2.00 to 25.59, while both the 
standard deviation and the standard error of the mean decreased slightly. At 1 .64, the t value was 
not significant, but the 0.367 effect size was “small” and the next to largest such value in Table 2. 



*Due to rounding, slight discrepancies might occur between the Mean and Mean Score Difference 
columns. 
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Table 2: 



Teacher Questionnaire Subscale t Tests and 
Effect Sizes Over Three Administrations 



Administration 


Number 




Standard 


Standard 


t 




Mean Score 


Effect 


Time 


of Pairs 


Mean 


Deviation 


Error of Mean 


Value 


Probability 


Difference 


Size 


Status 


Pretest 


27 


23.63 


3.61 


0.694 


2.22 


. 035 ° 


- 0.89 


-0.247** 


End of Institute 


22.74 


4.13 


0.795 










Pretest 


18 


23.72 


4.23 


0.996 


0.28 


.781 


- 0.28 


- 0 . 066 * 


Posttest 


23.44 


3.89 


0.821 










Professional Growth 


Pretest 




17.50 


1.93 


0.366 






- 0.14 


- 0 . 072 * 


28 








0.45 


.659 


End of Institute 




17.36 


1.75 


0.330 










Pretest 




17.33 


2.14 


0.505 








0 . 107 * 


18 






0.37 


.717 


0.22 


Posttest 




17.56 


1.58 


0.372 










Self-Efficacy 


Pretest 




49.46 


4.84 


0.950 




.719 




0 . 047 * 


26 








0.36 


0.23 


End of Institute 




49.69 


4.31 


0.844 










Pretest 


16 


48.75 


5.45 


1.362 


0.97 


.348 


1.38 


0 . 253 ** 


Posttest 


50.13 


4.54 


1.136 










Decision Making 


Pretest 


26 


24.69 


5.55 


1.088 


2.15 


. 042 a 


1.38 


0 . 250 ** 


End of Institute 


26.08 


5.20 


1.020 










Pretest 


17 


23.59 


5.47 


1.326 


1.64 


.120 


2.00 


0 . 367 ** 


Posttest 


25.59 


4.78 


1.160 










Impact 


Pretest 


26 


17.15 


3.59 


0.703 


0.00 


1.000 


0.00 


0.000 


End of Institute 


17.15 


3.77 


0.739 










Pretest 


17 


17.41 


3.54 


0.858 


1.31 


.210 


1.18 


0 . 334 ** 


Posttest 


18.59 


2.72 


0.659 










Autonomy in Scheduling 


Pretest 




8.26 


3.60 


0.693 








0 . 164 * 


27 








1.25 


.221 


0.59 


End of Institute 




8.85 


3.12 


0.601 










Pretest 




7.71 


3.75 


0.911 








0 . 376 ** 


17 




1.71 


.107 


1.42 


Posttest 




9.12 


2.91 


0.706 












*Effect Size = Less than small. 



a 
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**Effect Size = Small. 



'Significant at .05. 
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Table 3: 

Teacher Questionnaire Total Score t Tests and Effect Sizes 
Over Three Administrations 



Administration 

Time 


Number 
of Pairs 


Mean 


Standard 

Deviation 


Standard 
Error of Mean 


t 

Value 


Probability 


Mean Score 
Difference 


Effect 

Size 




Total Scale Score 






Pretest 


23 


139.39 


16.50 


3.39 


0.82 


.422 


1.48 


0.091* 


End of Institute 




140.87 


15.42 


3.22 










Pretest 


14 


135.29 


17.34 


4.63 


1.65 


.122 


8.43 


0.486** 


Posttest 




143.71 


11.70 


3.13 











*Effect Size = Less than small. 
**Effect Size = Small. 
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The Impact subscale results are displayed next in Table 2. With a possible score from 
5 to 25 points, the first t test reported no change in scores — up or down. With no score 
difference, there was no t value nor effect size. For the second t test on the Impact subscale, there 
was a 1.18 increase in the mean score to 18.59 at posttest. Both the standard deviation and 
standard error of the mean declined slightly. The 1 .3 1 / value was not significant, but the 0.334 
effect size was “small.” 

The last subscale in Table 2 is Autonomy in Scheduling with the possible score ranging 
from 3 to 15 points. The mean score in the first t test increased by 0.59 to 8.85, with a standard 
deviation of 3. 12. The 1 .25 t value was not significant, while the 0. 1 64 effect size was “less than 
small.” The second t test showed a 1.42 increase in the mean, with a standard deviation of 2.91 
and a standard error of the mean of 0.706. At 1 .71 , the t value was not significant. However, 
the pretest to posttest effect size of 0.376 is the largest such value in Table 2, although, according 
to Cohen’s terms, it is labeled “small.” 

Table 3 displays the results of the three administrations of the Teacher Questionnaire in 
terms of the Total Score. The possible score could range from 38 to 190 points. For the 23 
participants with complete data, the pretest mean increased 1.48 at the end-of-institute 
administration. The standard deviation declined more than 1 point and the standard error of the 
mean declined slightly. At 0.82, the t value was not significant and, at 0.091 , the effect size was 
much “less than small.” The pretest to posttest mean score rose 8.43 points to 143.71 , while the 
standard deviation declined nearly 6 points to 1 1 .70. The / value of 1 .65 was not significant. 
However, the effect size of 0.486, the largest such value for the Teacher Questionnaire, was 
labeled “small” but is very close to the lowest value for an effect size to be labeled “medium” by 
Cohen, which is 0.500. 



School As Community Questionnaire 

Table 4 Displays the results of the three administrations (pretest, end-of-institute, and 
posttest) of the School as Community Questionnaire to the institute participants. All five sub- 
scales and the Total Score results are presented in Table 4. The number of participants with both 
scores for the t tests varied from 1 8 to 27 for the subscales and was 24 and 1 8 for the Total Score. 

The Shared Sense of Purpose subscale results are displayed first in Table 4. The possible 
subscale score could range from 3 to 15 points. The pretest mean was 9.89 and the standard 
deviation was 1.81. The end-of-institute mean increased 0.22 to 10.11, while the standard 
deviation increased to 2.46 for the 27 participants. The standard error of the mean also increased 
to 0.472. The resultant t value was a low 0.69, which was not close to being significant. The 
effect size was 0.122, which was “less than small” in Cohen’s terms. The pretest to posttest 
scores showed a very small increase of 0.05 to 10.32, while the standard deviation increased 0.59 
to 2.69. The standard error of the mean also increased to 0.6 1 7. The t value was extremely small 
at 0.09 and was close to unity. The effect size of 0.029 was much “less than small.” 
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School as Community Questionnaire Total Scale and 
Subscale t_ Tests and Effect Sizes Over Three Administrations 



Administration 

Time 


Number 
of Pairs 


Mean 


Standard 

Deviation 


Standard 
Error of Mean 


t 

Value 


Probability 


Mean Score 
Difference 


Effect 

Size 


Shared Sense of Purpose 






Pretest 




9.89 


1.81 


0.347 










End of Institute 


27 


10.11 


2.46 


0.472 


0.69 


.496 


0.22 


0.122* 


Pretest 




10.26 


2.10 


0.483 










Posttest 


19 


10.32 


2.69 


0.617 


0.09 


.932 


0.05 


0.029* 




Collaborative Activity 




Pretest 




15.38 


4.47 


0.877 












26 








3.11 


.005 a 


2.38 


0.534*** 


End of Institute 




17.77 


4.17 


0.818 










Pretest 




16.00 


4.33 


0.994 












19 








1.86 


,079 c 


2.11 


0.487** 


Posttest 




18.11 


4.37 


1.003 










Collective Focus on Student Learning 




Pretest 




14.62 


2.35 


0.461 










End of Institute 


26 


14.96 


2.41 


0.472 


0.63 


.534 


0.34 


0.145* 


Pretest 




15.32 


2.24 


0.513 












19 








0.70 


.494 


0.53 




Posttest 




15.84 


2.39 


0.548 








0.232** 


Deprivatized Practice 




Pretest 




6.54 


2.58 


0.506 










End of Institute 


26 


6.69 


2.54 


0.499 


0.45 


.655 


0.15 


0.058* 


Pretest 




6.94 


2.90 


0.683 












18 








0.00 


1.000 


0.00 


0.000 


Posttest 




6.94 


2.44 


0.574 










Reflective Dialogue 




Pretest 




18.00 


5.16 


1.012 










End of Institute 


26 


18.88 


4.79 


0.939 


1.51 


.143 


0.88 


0.171* 


Pretest 




19.05 


3.92 


0.900 












19 








0.39 


.698 


-0.42 


-0.107* 


Posttest 




18.63 


3.32 


0.762 










Total Scale Score 




Pretest 




64.83 


13.48 


2.752 










End of Institute 


24 


68.79 


13.82 


2.821 


2.09 


.048 b 


3.96 


0.294** 


Pretest 




67.22 


12.37 


2.916 










Posttest 


18 


66.78 


17.15 


4.042 


0.09 


.927 


-0.44 


-0.036* 



♦Effect Size=Less than small. **Effect Size=Small. ***Effect Size=Medium. 
0 Significant at .01. b Significant at .05. c Si®\ificant at .10. 
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The Collaborative Activity subscale results are displayed second in Table 4. With a possible 
score ranging from 6 to 30 points, the pretest mean increased 2.38 to 1 7.77 at end-of-institute. This 
was the largest increase across the five subscales in the table. The standard deviation decreased 
slightly to 4. 17 and the standard error of the mean also decreased slightly. The resulting t value was 
3.1 1, the largest in the table. This t value was significant at the .01 level, the only one in the table 
significant at that level. At 0.534, the effect size was “medium” and was the largest in Table 4. The 
pretest to posttest score increased 2. 1 1 for the 1 9 participants, which was the second largest increase 
across the subscales. Also, both the standard deviation and the standard error of the mean increased 
slightly, the latter the second largest such value for the subscales. The / value was 1 .86, which was 
significant at the .10 level. At 0.487, the effect size for the pretest to posttest subscale technically 
was “small,” but is very close to the 0.500 value needed to be labeled “medium” by Cohen. 

The Collective Focus on Student Learning subscale results are displayed next in Table 4 
with possible scores ranging from 4 to 20 points. The mean score in the first / test increased 0.34 
to 14.96, while the standard deviation also increased slightly (0.06) to 2.4 1 . The resulting / value 
of 0.63 was not close to being significant and the 0.145 effect size was “less than small.” For 
the 19 participants, their mean score increased 0.53 to 15.84, and the standard deviation increased 
slightly (0.1 5) to 2.39. The resulting / value of 0.70 was not close to being significant. However; 
the effect size of 0.232 was “small” for the pretest to posttest administration. 

The Deprivatized Practice subscale results are the fourth set of rows in Table 4. With a 
possible score of 3 to 15 points, all four scores were less than 7 points. The first t test mean 
increased 0. 15 to 6.69, whereas the standard deviation decreased slightly (0.04), as did the standard 
error of the mean (0.07). At 0.45, the / value was not significant and the 0.058 effect size was much 
“less than small.” Interestingly, the pretest to posttest scores were the same at 6.94, yielding no / 
value or effect size. The standard deviation and standard error of the mean both decreased 
somewhat. 

The last subscale in Table 4 is Reflective Dialogue with a possible score ranging from 6 to 
30 points. The mean score in the first t test increased 0.88 from 1 8.00, while the standard deviation 
dropped slightly to 4.79. The standard error of the mean also decreased from 1.012 (the largest such 
value in the table) to 0.939. The / value was 1 .5 1 , which was not significant. The effect size was 
0.171, which was “less than small.” Unexpectedly, the pretest to posttest mean dropped 0.42 to 
18.63 (although the pretest mean for the 19 participants was the highest for this subscale). Also, 
both the standard deviation and the standard error of the mean decreased at posttest. At 0.39, the 
t value was very small and nonsignificant. The effect size of -0.1 07 was “less than small.” 

The last set of rows in Table 4 display the results for the Total Score, which could range 
from 22 to 110 points. For the 24 participants, the pretest mean increased 3.96 to 68.79. Both 
the standard deviation and the standard error of the mean increased slightly. The resultant / value 
was 2.09, which was significant at the .05 level. The effect size of 0.294 was “small.” 
Unexpectedly, the pretest to posttest mean decreased 0.44 to 66.78. The standard deviation 
increased almost 5 points to 17.15. The standard error also increased over 1 point to 4.042. The 
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resultant / value was a miniscule 0.09 and was close to unity. The effect size of -0.036 was much 
“less than small.” 



School Products and Services Questionnaire 

Table 5 displays the results of pretest and posttest administrations of the School Products 
and Services Questionnaire. (This questionnaire was not planned to be administered at the end 
of the week-long institute.) The possible score on this instrument ranged from 8 to 40 and 18 
participants had both scores. The pretest mean was 28.06 and its standard deviation was 4.64. 
The standard error of the mean was 1.095. The posttest mean increased 0.39 to 28.44, whereas 
the standard deviation decreased to 4.30. The standard error of the mean also decreased very 
slightly. The resultant t value was a very small 0.47, which failed to approach significance. The 
effect size was 0.082, much “less than small” in Cohen’s scheme. 



Control Questionnaire 

Table 6 displays the results of the single administration of the Control Questionnaire. 
This instrument was administered to describe the institute participants on this construct, thus only 
descriptive statistics are presented in Table 6. There are two subscales and the Total Score for 
the Control Questionnaire, with complete data for 30 participants. 

The Self-Confidence subscale results are displayed first in Table 6. The possible subscale 
score could range from 14 to 70 points. Actual scores ranged from a low of 47 points to a high 
of 66 points. The mode was 61.00 and the median was 59.50. The mean was 57.43, with a 
standard deviation of 6.55. The coefficient of variation (the standard deviation divided by the 
mean) was a low 0. 1 1 4. The standard error of the mean on Self-Confidence was 1 . 1 95. 

The Autonomous Behavior subscale results are displayed next in Table 6. With the same 
number of possible points (14-70), the actual scores ranged from a low of 41 to a high of 65. The 
modal score was 50.00, 1 1 points lower than that for the first subscale. The median was 5 1 .50, 
8 points lower than that for the first subscale. At 52.03, the mean was 5.40 points less than Self- 
Confidence. Moreover, the standard deviation was approximately half a point less at 5.92. 
Interesting, the coefficient of variation value was exactly the same as above, although the standard 
error of the mean was a little smaller (1.081). 

The last section of Table 6 shows the Total Scale Score statistics for the Control 
Questionnaire. Here, the possible scale score could range from 28 to 140 points. The actual 
scores ranged from a low of 9 1 to a high of 1 3 1 , only 9 points from the maximum possible score. 
The modal score was 93.00, and the median was 1 1 1 .50. The mean was 1 09.47, with a standard 
deviation of 10.81. The coefficient of variation was just 0.099, the smallest of the three such 
values in the table. The standard error of the mean was 1 .974, the largest of the three in the table. 
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Table 5: 

School Products and Services Questionnaire t Test 
and Effect Size Over Two Administrations 
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Administration 

Time 


Number 
of Pairs 


Mean 


Standard 

Deviation 


Standard 
Error of Mean 


t 

Value 


Probability 


Mean Score 
Difference 


Effect 

Size 




Total Scale Score 




Pretest 


18 


28.06 


4.64 


1.095 


0.47 


.643 


0.39 


0.082* 


Posttest 




28.44 


4.30 


1.014 











*Effect Size = Less than small. 
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Table 6: 

Descriptive Statistics for the Control Questionnaire 



Number 

Respondents 


Minimum 


Maximum 


Mode 


Median 


Mean 


Standard 

Deviation 


Coefficient 

ofVariation 


Standard 
Error of Mean 




Self-Confidence 






30 


47.00 


66.00 


61.00 


59.50 


57.43 


6.55 


0.114 


1.195 




Autonomous Behavior 






30 


41.00 


65.00 


50.00 


51.50 


52.03 


5.92 


0.114 


1.081 




Total Scale Score 




30 


91.00 


131.00 


93.00 


111.50 


109.47 


10.81 


0.099 


1.974 
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Feedback Form 

The Summer Institute Feedback Form was administered on the last day to 29 participants: 
23 teachers, 5 administrators, and 1 intern. The overall internal consistency reliability was .74. 

The first 1 5 items asked respondents to rate the extent to which the Institute had met 1 5 
separate evaluative topics. The response option for each item was a scale of 1 (Not at all) to 5 
(Very much). Table 7 displays the response option frequencies and percentages, means, and 
standard deviations for all 1 5 items. None of the items received a rating below a 4 on the 5-point 
scale. In fact, 3 of the items (#7, #1 1, and #12 — all facilitator-related) received consistent ratings 
of 5 from all participants. Similarly, all 15 means were above 4.75, and standard deviations 
ranged only from 0.00 to 0.44. 

Question 1 6 asked how respondents would describe the Institute experience to others. Out 
of the 41 comments generated, almost half (18, 44%) described it as positive (or similar terms); 
i.e., “A well-planned, carefully orchestrated, flexible workshop.” The next frequent theme was 
motivating/rewarding (8, 22%), “It was inspiring and I’m ready to go to work!” Other themes- 
included informative (5, 12%), team building (4, 10%), and miscellaneous (5, 12%). One 
participant noted, “It ties together in a meaningful way the tasks for teaching and provides the 
tools to do the job.” 

In Question 1 7, participants were asked what ideas, strategies, activities, or resources were 
most meaningful. A total of 76 were provided. The most frequently mentioned topic was team 
building (15, 20%), followed by the rubrics (10, 13%). Other identified topics included 
assessment (9, 12%), resources (7, 9%), and the “types of intellect” hats (6, 8%). A variety of 
other responses were provided, including multiple intelligences, the “hands” activity, consensus 
activities, and community-based learning. 

Respondents were asked for major strengths of the Institute in Question 18. Of the 54 
comments provided, the most frequently mentioned strength was team building (14, 26%), 
followed by new information/ideas (1 1, 20%) and organization/implementation (10, 19%). One 
participant commented, “Constant modeling of how to do block time (I never looked at my 
watch).” Other strengths included presenters (8, 15%), activities (5, 9%), participants (4, 7%), 
and interns (2, 4%). 

Question #19 asked for recommendations for future institutes. Only 31 comments were 
generated for this question. Almost half (13, 42%) focused on adjusting the agenda or schedule. 
These recommendations ranged from giving participants more free time in the evenings to adding 
more evening activities, as well as adding more time for discussion. Six of the respondents (19%) 
said no changes were needed; in fact, one noted, “Hard to improve on perfection!” And, a few 
participants mentioned providing more examples/information, improving the facilities, and 
emphasizing team building. 
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Table 7: 

Participants’ Responses to 15 Evaluative Topics 
on the End of Institute Feedback Form (N = 29) 



I to ID s 


Response Option Frequencies and Percentages 


Mean 


SD 




Not at all 
1 


2 


3 


4 


Very much 

5 


1. Had clear outcomes. 




- 


- 


2 

6.9% 


27 

93.1% 


4.93 


0.26 


2. Promoted team building. 




- 


- 


1 

3.4% 


28 

96.6% 


4.97 


0.19 


3. Caused me to reflect on my 
practices. 




- 


- 


4 

13.8% 


25 

86.2% 


4.86 


0.35 


4. Caused me to examine some 
of my attitudes. 


- 


~ 


- 


7 

24.1% 


22 

75.9% 


4.76 


0.44 


5. Facilitated development of 
new skills. 


- 




- 


1 

3.4% 


28 

96.6% 


4.97 


0.19 


6. Was relevant to my needs. 


- 


- 


- 


5 

17.2% 


24 

82.8% 


4.83 


0.38 


7. Was conducted in a positive 
manner. 


- 


- 


- 


- 


29 

100.0% 


5.00 


0.00 


8. Had activities that were well 
sequenced.* 


- 


- 


- 


4 

14.3% 


24 

85.7% 


4.86 


0.36 


9. Had activities that rein- 
forced content. 




- 


- 


5 

17.2% 


24 

82.8% 


4.83 


0.38 


10. Included appropriate 
examples. 


- 


- 


- 


4 

13.8% 


25 

86.2% 


4.86 


0.35 


1 1 . Was conducted by 
competent trainers. 


- 


- 


- 


- 


29 

100.0% 


5.00 


0.00 


12. Was conducted in a 
professional manner. 


- 


- 


- 


- 


29 

100.0% 


5.00 


0.00 


13. Had meaningful involve- 
ment of participants. 


- 


- 


- 


2 

6.9% 


27 

93.1% 


4.93 


0.26 


14. Stimulated me to use the 
concepts/skills/materials. 


- 


- 


- 


3 

10.3% 


26 

89.7% 


4.90 


0.31 


15. Provided materials/ideas 
that will be useful to me. 


- 


- 


- 


1 

3.4% 


28 

96.6% 


4.97 


0.19 



*7/= 28. 
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In Question 20, participants were asked how the Institute had changed the way they 
thought about ITI. Of the 28 comments, 8 (29%) indicated a positive change, and 5 participants 
each (1 8%) mentioned a new belief or reinforcement of ITI ideas and seeing the “big” picture of 
what ITI comprises. Representative comments include “I was skeptical before; now I see it can 
be done and actually work” and “I now have the full/whole picture and will be able to share with 
others.” Other changes were seeing the possibilities of using ITI (4, 14%), their enthusiasm to 
implement ITI (3, 1 1%), and the potential of using ITI in secondary schools (3, 1 1%). 

Respondents were asked how they would use what they learned in Question 21 . Of the 
49 comments, 13 (27%) mentioned incorporating ideas and activities (unspecified) in the 
classroom (“I will change my methods of teaching”), while another 13 focused on team-building 
with teachers (“More teams in classroom and hopefully more teams with teachers across the 
curriculum”). Eleven of the comments (22%) identified specific activities to incorporate, such 
as rubrics, problem-solving, hats, cooperative learning, multiple intelligences, assessment, and 
journaling. Other themes included sharing information with colleagues and staff development 
uses (9, 18%) and adding student-based teams in the classroom (3, 6%). 

Question 22 asked respondents how AEL could continue supporting curriculum 
integration efforts. Of the 37 comments, the two main suggestions were keeping in touch with 
participants (12, 32%) and providing new information and resources as available (11, 30%). 
Representative comments include “Just ‘be there’ for us — we will need your support” and “Keep 
up the outstanding work — this needs to be implemented in all of our schools.” Other suggestions 
included followup sessions (5, 14%), internet or e-mail communications (4, 11%), and 
miscellaneous ideas (5, 14%). 

Finally, Question 23 asked for other comments. Eleven of the 36 comments (31%) were 
congratulatory. Examples include “Super — great institute!!! Of my 25 years of educational 
workshops, this one ranks the highest on the Rubric Assessment”; “This has been time and money 
well spent”; and “In the end — closure does bring those connections and products that we came 
for.” Nine of the comments (25%) mentioned positive aspects of the facilities, while 7 (19%) 
focused on negative aspects. And, 7 comments (19%) were simply “thank-you’s.” 



Interviews 

The AEL evaluator conducted individual interviews with 29 participants at some point 
during the Institute — during breaks, lunches, or evening hours. Interview results were hand- 
written on summary sheets, which formed the basis for this analysis. 

The interviewer first asked how much decision-making authority the respondent had over 
what and how material was taught. Seven of the 29 respondents felt they had total control in 
deciding what was taught. In an interesting dichotomy, 6 more respondents noted they had total 
control over what was taught, as long as it was within state guidelines/curriculum — while 5 others 
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perceived they had no control over what was taught, indicating it was based on state guidelines. 
While the underlying premise seems to be the same for both viewpoints, there seems to be great 
discrepancy in participants’ perceptions of that premise and how it relates to them. Six others 
said they had a great deal of freedom in selecting what to teach, and an additional 3 said they had 
limited authority. Two administrators indicated these decisions were made by teams or by 
department chairs. When reporting how much authority they had in deciding how material was 
taught, the majority (17) said they had total control in this area, 7 said they had a great deal of 
control, 2 said they had limited authority, and 1 indicated no authority in this area. Two 
administrators indicated they would not dictate how material was taught. 

The second interview question asked respondents to describe ways they were helping 
students to become independent learners; 46 ways were provided. Critical thinking/analytical skills 
was most frequently mentioned, by 9 of the 29 participants. The second topic was activities and/or 
projects, mentioned by 5 participants. Four respondents each described cooperative learning, 
individualized learning, and computer training. Other teacher-focused responses included profes- 
sional development, outside resources, questioning techniques, mastery teaching, and modeling. 
Other student-centered responses included self-evaluation, small groups, discussions, writing, 
research, self-esteem, and personal responsibility. Sample comments include: “I try to incorporate 
activities and skills that they will need in life, especially critical thinking skills.” “I return questions 
with questions to students. I work hard on developing critical thinking in students.” “I try to 
provide activities that provoke students to want to learn more.” “This is my goal, to make them 
independent. I stress logical consequences of what they do.” “I have them research answers. 

I make them look up words, answers, etc. I try to get away from giving them the answers.” 

Next, participants were asked howtheir team members would describe students’ learning. 
Of the 29 respondents, almost half (13) felt their team members would say student learning was 
positive or average. One commented, “I think they would say that our students are learning, but 
they’re interested in enhancing learning more.” Another 8 felt team members would say that 
learning differs by students, according to their individual abilities. As 1 respondent noted, “They 
would say that our kids are learning just like average American kids, some high, some low.” Two 
participants felt their team members would say students were not reaching their potential, and 6 
others said it was too early to determine yet. 

For the fourth question, the evaluator asked participants how they would judge their school 
team at collaborating on professional-type issues. Almost half (1 2) of the participants judged their 
team’s collaboration to be excellent or very good. For example, “We have a very professional-type 
team; excellent” and “Very positive, very professional, very intense, self-motivated; very 
progressive.” Seven judged the collaboration to be good; i.e., “Good, everybody expresses their 
point and we reach agreement” and “Our team does a good job at collaborating on professional 
issues.” Four other participants felt their teams were learning to collaborate: “Working at it, that’s 
why we’re here” and “We became a team to come here to leam ITI.” One respondent noted the 
school team collaborated “with difficulty,” while another commented “Rather early to tell on this, 
it’s still overwhelming to us.” Three others said it was too early to determine yet. 
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Last, the evaluator asked participants to rate their school faculty on readiness and willing- 
ness to accept change. Almost half of the respondents (13) noted it would be difficult, citing 
reasons such as “old” faculties, constantly hearing about new things, lack of unity, negative 
attitudes, skepticism, and resistance. Seven participants felt about half of their faculty would 
accept changes, with several noting that their schools had implemented so many changes lately 
that it may impact teachers’ willingness to try anything else. Alternatively, 7 other participants 
felt the majority of their faculty were ready and willing to implement school change. Finally, 2 
others said it was too early to determine yet. 



Role-Alike Caucus Sheet Reflective Activity 

For this activity, participants were to assemble in role-alike groups. The six role groups 
included administration, English/language arts, mathematics, science, social studies, and other 
(including family /consumer science, foreign language, marketing, music, and vocational 
education). For each group, participants were to select a facilitator, a timekeeper, and a recorder, 
and were to answer two questions. 

The first question was, “What to you seems most promising by way of new approaches 
to teaching and learning?” The administration group’s responses included programs that reach 
all learners and help students become active learners; learning styles and strategies; and the 
emphasis on skills in computers, communication, reading, and higher-level thinking. The English 
group mentioned electronic media, learning styles, block scheduling, group learning, curriculum, 
portfolios, and outcome-based education. The math group responded with school to work, 
applied subjects, and questioning techniques. The science group mentioned utilizing technology 
and school to work, showing students how school relates to the real world, and students’ higher 
interest level. The social studies group’s responses included transfer of learning to students, 
shared responsibilities, problem solving and communication skills, and cooperative learning. 
Finally, the other group mentioned technology, increased student interest and teacher knowledge, 
higher student success rates, hands-on learning, and learning to apply to daily life. Looking at 
responses across roles, five of the groups mentioned learning styles and strategies (including 
higher-level thinking, group learning, questioning techniques, cooperative learning, and hands-on 
learning), and four groups mentioned utilizing new technology. Three groups mentioned 
increasing student interest and participation, and two groups each mentioned applying learning 
to daily life/real world, utilizing school to work, and improving communication skills. 

The second question was, “What special opportunities and challenges are associated with 
collaborating with others in designing interdisciplinary studies?” The administration group 
mentioned getting teachers to become risk takers, giving the time/resources/opportunities to risk 
takers, setting higher expectations for teachers and self, and actually teaching the students (not 
just the subject matter). The English group’s responses included specific challenges (egos, fear, 
reluctance to change, and scheduling) and opportunities (shared support/ideas, teacher 
development, and reduced burnout). The math group mentioned time to collaborate, prioritizing 
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educational changes, support of vocational technical program, sharing prior integration 
experiences, and learning connections between classroom and life skills. The science group 
mentioned one opportunity (tying things together for students) and several challenges (teachers 
isolated in “discipline of student,” scheduling/school structure, meeting T-CAP scores and college 
requirements, and “catch-22” situations). The social studies group’s responses included 
willingness to compromise, team challenge to work together, and changing from an independent 
to an interdependent teacher. Finally, the other group mentioned opportunities of pooling 
resources, exchanging ideas, meeting other teachers and people out of your particular area, and 
seeing a greater success rate; and challenges of coming out of the “comfort zone,” overcoming 
administrative attitudes, finding time for meetings, budgeting issues, and educating parents and 
the community. Looking at responses across groups, several issues seem to be prevalent. Similar 
challenges included time restraints, scheduling and structure issues, and persuading staff to 
become risk takers. Similar opportunities included collaboration, networking, and sharing 
resources and ideas. 



Team Reflection Logs 

In this activity, the six teams were asked to complete a Team Reflection Log at the close 
of each day’s sessions (except for the last day). The items for each log were tailored to the 
content of the day’s events. The sixth team, composed of interns, only completed a log for 
Sunday, which is not included in the following analysis. 

Sunday. This daily log asked each team for the following information: Which of today’s 
activities helped set the stage for work at this Institute? What can we do/provide/change to better 
meet your team needs and interests? What are your team’s “next steps” in designing interdisci- 
plinary teamed instruction? Four adjectives to describe our feelings. Other things we’d like to say. 

Team A (school names deleted to provide anonymity) mentioned “team strengths through 
hands” when responding to which of the activities helped set the stage for work at the Institute. 
Team B indicated “roles for table talk, hands poster, and the journal.” Team C also mentioned 
“hands” and “meeting with others in discipline areas.” Team D again mentioned “the hands 
activity,” as well as “spelling out tasks to be done” and “organized presentations.” Finally, Team 
E also mentioned the “hands activity” and “caucus (concerning subject areas).” Clearly, the 
“hands” activity seemed to be the definitive activity to begin the Institute. 

When asked what could be done to better meet team needs and interests, Team A 
indicated a need for explanations on the “Role Play Intg. Lesson,” while Team B simply 
responded with “more time for transition.” Team C said to “provide practical ideas and 
suggestions to implement integrated curriculum as easily and smoothly as possible.” Team D had 
one comment (“too much, too fast, to think about”) and two suggestions: “more sharing time in 
group” and “need answers to questions— how do we make it work?” Finally, Team E asked for 
“two copies of each handout (one for recorder, one for facilitator).” 
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The third item asked teams what their “next steps” were in designing interdisciplinary 
teamed instruction. Team A responded, “learning how to design and plan a lesson,” while Team 
B said “active participation by all team members, more comfortable atmosphere, we feel this 
question is vague.” Team C said “network with other teams, gathering information, gain a vision 
of where we want to go.” Team D replied, “We don’t know yet — we need to see models of 
successful ITI.” And, Team E mentioned “implement strategies you teach us, find out pilot area 
for integration for our school.” 

In providing adjectives to describe their feelings, Team A listed “tired, dubious, 
apprehensive, major concern”; Team B, “negative, dedicated, open to ideas, flexible”; Team C, 
“tired, expectant, excited, anticipation”; Team D, “confused, frustrated, overwhelmed, curious”; 
and Team E, “expectant, hopeful, concerned, overwhelmed.” Obviously, these five teams were 
similar in their feelings of being a little wary of what was taking place, while at the same time 
trying to stay open-minded and positive about the outcome. 

Finally, when asked for other comments, Team A replied, “We are not negative. We are 
realistic about problems we have no control over to implement the program.” Team B asked for 
“more literal questions, more clarification.” Team C said, “This session was a great kick-off. It 
was well-paced. The leaders are easy to relate to/approach.” Similarly, Team D replied, “You 
are very organized. You seem very knowledgeable].” Team E did not provide comments for this 
item. 



Monday. Items for Monday included: Which of today’s activities best helped you 
prepare for your team role and responsibilities at the Institute? What can we do/provide/change 
to better meet your team needs and interests? What are your team’s “next steps” in developing 
your capacity for teamwork? Four verbs to capture our feelings. Other things we’d like to say. 

Team A indicated two activities that best helped them prepare for their team role and 
responsibilities: “Six hats helped us to see what type of thinkers we were. Technology (car 
building) each team member faced their own weaknesses.” Teams B, D, and E responded 
similarly with the hats and mousetrap-powered vehicle activities. Team E added, “The hats 
helped to show us what talents the group had; however, we did not use that info before making 
a car — the car was a disaster.” Team C also listed the mousetrap activity, as well as “Tools to 
Support Teamwork.” 

When asked what could be done to better meet team needs and interests, Team A replied, 
“Meet with our team to discuss our direction.” Team B suggested, “Check for understanding after 
directions are given. Clarify directions/questions.” Team C responded, “Keep on keeping on — 
you’re great. Have the high schools share their implementation w/ us for our 9 th grade 
possibilities.” Team D replied, “Realign agenda — possibly work until 5 and free up time after 
supper for team planning, thinking, reflection, reading, and interchange with other teams.” And, 
Team E said, “Enough R&Rto go and explore Radford. Maybe rearrange schedule — cut out p.m. 
breaks, go longer and get off at 4 or 5.” 
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In identifying their next steps in developing teamwork capacity, Team A said, “We are 
implementing our Table Top Rules to be more successful in becoming an expert team.” Team 
B indicated a need to “clarify roles/willingness to assume roles.” Team C said they would “lay 
the groundwork for an 8 th grade team,” while Team D mentioned “collaborating on possible 
projects for here and at school.” And, Team E replied, “coming up w/ ideas of units we can use 
to integrate curriculum.” 

When asked to list four verbs that captured their feelings, Team A listed “agitated, 
excited, questioning, vocal”; Team B, “produce, create, develop, talk/listen”; Team C, “doing! 
thinking, reevaluating, reflecting”; Team D, “better, tired, processing, evaluating”; and Team E, 
“planning, questioning, brainstorming, collaborating, wondering, pondering.” Based on these 
descriptors, it seems that all teams were actively involved in collaborative development activities. 

Finally, when asked for other comments, Team A said, “Thank God for Mr. Merrill!!! 
How does all of this come together?” Team B replied, “Joshua was enlightening/thought- 
provoking. Time is still too short sometimes. Great organization on your part!!!” Team C 
simply replied, “Thanks! Good day!” Team D said, “Please consider our request for after supper 
work.” And, Team E replied, “This is frustrating. Beginning to see glimmer of what is possible. 
Beginning to hope for attitude changes on part of our faculty.” 

Tuesday. The items included: Which of today’s activities best prepared you to make 
decisions about interdisciplinary teamed instruction? What can we do/provide/change to better meet 
your team needs and interests? What are your team’s “next steps” in terms of designing an 
interdisciplinary course or unit? Four nouns that sum up our feelings. Other things we have to say. 

Team A provided three activities that best prepared them to make decisions about ITI: 
“Jigsaw ITI examples, reviewing the examples, reviewing fertile themes for integrated learning.” 
Team B also mentioned the jigsaw, as well as the “Red, Blue, Green, and Yellow” article. Team 
C responded similarly with “reading ‘Selecting Fertile Themes,’ reviewing and rating ITI 
examples, designing our individual unit.” Team D replied, “mtn climbing. — 4 corners activity,” 
while Team E also mentioned the “fertile themes” article. Overall, this article seemed valued by 
most teams. 

When asked what was needed to better meet team needs and interests, Team A replied, 
“We appreciated individual help today.” Team C said to “continue similar activities/provisions. 
Team D responded, “Thanks for rearranging the schedule to consider our requests.” And, Team 
E indicated, “You are all doing great — you are here when we need you — and you gave us an 
R&RH” 



In describing next steps for designing an interdisciplinary course or unit, Team A 
indicated “designing activities and curriculum”; Team B, “children input”; Team C, “focusing 
on specific activities for our unit”; Team D, “selecting the disciplines to include in our unit”; and 
Team E, “We are going to refine and polish our unit (and probably change it) and add activities.” 
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In listing four nouns that summed up team feelings, Team A said “understanding, 
guidelines, direction, success”; Team B, “team, enthusiasm, energy, thoughts”; Team C, 
“thinkers, workers, creators, planners”; Team D, “themes, questions, concepts, lens, energy”; and 
Team E, “jubilation, accomplishment, work, taskmaster.” Again, all teams seemed to be “in 
gear” as the creative process continued. 

When asked for other comments, Teams A and D said they were “beginning to see the 
light,” while Team B said “thanks for the evening off.” Team C noted “we see progress today” 
and Team E replied, “You, by the way you run the workshop into 90 minute blocks of time, have 
done a wonderful job of modeling how to efficiently use the time on block scheduling. You have 
changed activities often enough to keep our interest.” 

Wednesday. These items included: Which of today’s activities best helped you define 
active learning and authentic tasks? What can we do/provide/change to better meet your team 
needs and interests? What are your team’s “next steps” in terms of designing challenging and 
engaging tasks? Four words to describe our feelings today. Other things we have to say. 

When asked which activities best helped them define active learning and authentic tasks, 
Teams A and E mentioned the four concurrent sessions of “Writing to Learn, Community Based 
Learning, Cooperative Learning, and Multiple Intelligences.” Team B said the “Mrs. Tolliver” 
video, and Team C said, “The concurrent sessions were well presented and beneficial. Mrs. 
Tolliver’s class video was inspiring and ‘spotlight on a right’ all encompassing and thought 
provoking.” Similarly, Team D replied, “The videos helped us to integrate the activities of the 
conference. The breakouts helped with planning activities.” 

When asked what was needed to better meet team needs and interests, Team B asked for 
“consistent guidelines” and Team C noted that “just a few more minutes was needed on the 
concurrent session.” Team D replied, “Need time (10-20 minutes) to jigsaw report on breakout 
activities — immediately after breakouts — while info is fresh and we are still excited!” And, Team 
E said that “our needs are being met.” 

In identifying next steps for designing challenging and engaging tasks, Team A mentioned 
“reflect on theme — revise and refine as needed.” Team B listed “student input, group planning,” 
while Team C said “fine tune our project.” Team D noted a need to “review closely today’s plans 
to develop the tasks” and Team E said to “design staff development in int. curriculum for 
faculty.” 



In providing words that described their feelings, Team A said “relieved, tired, saturated, 
busy”; Team B, “tired, confused, informed, grateful”; Team C, “accomplishment, busy! ! ! , taxing, 
progress”; Team D, “touche, energetic, progressing, goals are clearer”; and Team E, “relieved, 
educated, productive, exhausted.” While team energy seemed to be waning, each team was close 
to a final product. 
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Other comments included Team A’s “Thanks for the help.” Team B said, “We are 
concerned about the exhibition on Friday: (a) time, (b) quality, (c) depth, (d) format.” Team C 
succinctly noted, “We are family.” Team D said, “We appreciated the videos. I can almost feel 
that ribbon snap across my chest as I cross the finish line!” And, Team E summed up with, 
“Thank you for listening to us. And for being positive model teachers.” 

Thursday. These items included: Which of today’s activities best helped you connect 
learning and assessment? What can we do/provide/change to better meet your team needs and 
interests? What are your team’s “next steps” in terms of building assessment into instruction 
rather than adding it on? Our rubric for today’s program is. Other things we’d like to say. 

When asked which activities best helped the team connect learning and assessment, Team 
A replied, “creating our own rubrics.” Team B said, “hands-on practice, thanks!” Team C noted 
the “rubric construction” and Team D said “making the rubric/assessment.” Team E replied, 
“Practicing magic squares and making a rubric on it. The original rubrics with combined groups 
gave our team a different and enlightened perspective on assessment.” Obviously, the rubric 
activities helped all teams connect the concept of learning with assessment. 

In responding to the item asking what could be done to better meet team needs and 
interests, Team A felt the “topic was thoroughly covered — very helpful.” Team B excitedly 
noted, “You’ve done it!! Great stimulation and activities!” Team C asked for “copies [of] units, 
list names, . . . Isosceles song.” Team D replied, “We (adults) need to be reminded to put our 
‘ego’ in check when dealing with teams and diverse opinions.” And, Team E, commenting on 
meeting facilities, said, “In the future have copy machine in room and use permanent overhead 
markers — others smear.” 

In describing next steps for building assessment into instruction (rather than adding it on), 
Team A said, “We want to plan to use more authentic assessment rubrics and less traditional 
methods. Those who do use authentic assessment want to refine rubric making techniques.” 
Team B noted, “After administering the initial questionnaire to students in Sept., we plan to have 
students generate guidelines for the interviews of family members.” Team C said they would 
“continue work at home with faculty and other team members [and] try to integrate unit with 
other teachers and integrate the assessment.” T earn D’ s next steps were to “reevaluate task, adjust 
task, agree.” And, Team E said, “Have students develop their own rubrics and use several 
alternative assessments (w/ rubrics) in each grading period.” 

When describing their rubric, Team A said, “rad, so-so, boring”; Team B said “we 
expected energetic presentation and we got DYNAMIC!”; Team C rated their rubric an 8 on a 9- 
point scale; Team D gave themselves an “A”; and Team E drew a lit-up light bulb, with the 
phrase, “the light dawned.” Although four of the teams seemed delighted with their final rubrics, 
Team A didn’t seem to be quite as pleased. 
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In providing other comments, Team A said, “super-great session, finally coming together, 
one of the most useful sessions.” Team B noted, “It was a fantastic week. We appreciate all your 
efforts.” Team C replied, “We are thankful for your hard work!” and Team D simply said 
“Thanks!!” Finally, Team E jokingly replied, “Pray for us not being lynched by rest of faculty. 
Thanks, you’ve been great.” 

Videotaped Presentations 

All six teams developed final presentations and presented them on the final morning of 
the Institute. These “culminating exhibitions” were to capture and convey three key concepts: 
(1) Reflect on your learnings and experiences at this Institute; (2) What key messages will you 
bring back to share with colleagues; and (3) How has your work together this week put a mark 
on how you view your role as teacher in the learning process? Teams could select one of the 
following themes for their presentation: TV news broadcast, TV sitcom, home page on the 
internet, TV commercial, TV/radio talk show, magazine, courtroom trial, game show, mystery' 
theater, or broadway musical. 

Each team’s videotaped presentation was reviewed by evaluators and assessed using a 
“Holistic Rubric Based on Clustered Activities.” This rubric included four levels: 

1. Novice: Some of the team members cannot be heard or seen well. There are gaps or 
pauses in the flow of the performance. Technologies or props may be lacking or even 
distracting. The whole program may appear not organized or is too long or too short. 

2. Apprentice: The message is mixed or missing, or participation is not balanced. Some 
elements may actually be done well, but this performance is distinguished by its 
unevenness. 

3. Practitioner: The intended message reaches the audience. Everyone can be seen and 
heard. Props and related technologies enhance the message without intruding on or 
overshadowing it. 

4. Expert: An exceptional performance. The story line and characters reflect a very high 
level of creativity, insight, or humor. The major messages developed at the week-long 
Institute are clearly conveyed. 

The six teams’ culminating exhibitions varied widely in their format, length, and quality, 
although they were uniformly entertaining and obviously fun for both presenters and the audience. 
In terms of the format for the team presentations, two were broadway musicals, one was a TV 
musical (because of the addition of an ITI commercial), one was a TV news broadcast, one was 
a TV shopping network, and one was a TV game show (“The ITI Feud”). The shortest 
presentation was just 2 minutes 5 seconds, while the longest was 13 minutes. All of the 
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presentations included some type of visual aid or prop such as handmade signs, pieces of a large 
puzzle, large maps, teaching aids, or overhead transparencies. The presentations varied in their 
“smoothness,” probably reflecting more or less rehearsal time or lack of experience in acting 
and/or singing. However, the chosen format, timing, or quality appeared to not hinder the 
audience from enjoying some teams’ presentations and joining in singing songs or repeating 
verses when requested. 

Charged with conveying three key concepts in the presentation, the six teams selected a 
variety of concepts, but there was remarkable similarity with one concept. Specifically, the 
concept of multiple intelligences was included in five of the six team presentations, and it 
dominated the time for two of those teams. Most teams used different colored hats to reflect 
different intelligences; either real hats or those made of construction paper or drawn on overhead 
transparencies. Four concepts were mentioned or portrayed in at least two presentations: 
teamwork, interdisciplinary units, assessment/rubrics, and facilitation roles. Other concepts in 
the presentations included “table talk” rules, “hands of traits,” transfer of learning, thinking of 
teaching and learning differently, and elements of a good teaching unit. The evaluators could not 
identify a third key concept in one team presentation, which was the lowest rated exhibition. 

Of the six presentations, two were judged to be at the Expert level, three at the Practitioner 
level, and one at the Apprentice level. Overall, team members seemed to be interested in and 
knowledgeable of the content of the 1996 ITI Summer Institute. 
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CONCLUSIONS 



A number of conclusions can be drawn from the findings presented earlier. These 
conclusions are presented by type of data collection, followed by overall conclusions. 

Teacher Questionnaire. We conclude that the 1996 ITI Summer Institute had some 
effect on the Self-Efficacy, Decision Making, Impact, and Autonomy in Scheduling subscales 
after a period of 1 year. For the Total Scale, the Institute seems to have had some effect, which 
was almost moderate after 1 year. The Institute had little effect on the Status and Professional 
Growth subscales after 1 year. 

School as Community Questionnaire. We conclude that the 1 996 ITI Summer Institute 
had a moderate effect on the Collaborative Activity subscale and some effect on the Total Score 
by the end of the Institute. The Institute had some effect on the Collaborative Activity and 
Collective Focus on Student Learning subscales after a period of 1 year. The Institute had little 
effect on the Shared Sense of Purpose and Reflective Dialogue subscales and the Total Score after 
1 year. The Institute had no effect on the Deprivatized Practice subscale after 1 year. 

School Products and Services Questionnaire. We conclude that the 1996 ITI Summer 
Institute had little effect on the Total Score after a period of 1 year. 

Control Questionnaire. We conclude that the 1996 ITI Summer Institute teachers scored 
higher on the Self-Confidence subscale in comparison to the Autonomous Behavior subscale. 
And, when the subscales were combined, the teachers scored moderately high on the Total Score. 

Feedback Form. We conclude that participants felt the 1996 ITI Summer Institute 
greatly met its intended outcomes. They found the Institute to be well planned, informative, 
motivational/inspirational, and worth the time and money invested. The most meaningful 
Institute activities for participants were team building, assessment rubrics, and multiple 
intelligences; major strengths were team building, information provided, and the organization/ 
implementation. The Institute positively changed participants’ thinking, resulting in increased 
belief in and reinforcement of ITI principles, and heightened awareness of its potential uses. 
Furthermore, participants were eager to incorporate their new knowledge back in their respective 
schools via rubrics, multiple intelligences, cooperative learning, and team building. 

Role-Alike Caucus Activity. We conclude that participants of the 1 996 ITI Summer 
Institute felt the most promising new approaches to teaching and learning included technology 
and learning styles and strategies. These styles and strategies included higher-level thinking, 
questioning techniques, group learning, cooperative learning, and hands-on learning. We also 
conclude that collaborating to design interdisciplinary studies brings about opportunities to those 
involved including collaboration, networking, and sharing new resources and ideas with others. 
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And, conversely, this collaboration invokes challenges such as time restrictions, scheduling 
problems, and persuading others to become risk takers. 

Interviews. We conclude that most participants of the 1996 ITI Summer Institute felt a 
great deal of freedom in deciding what and how curriculum is taught in their respective schools. 
Participants also advocated using strategies such as critical thinking and analysis skills, activities 
and projects, cooperative learning, and computer training to help students become independent 
learners. The majority of Institute participants described student learning in their respective 
schools as positive or average, with individual achievement levels differing by student. Most 
teams felt that their level of collaboration was good, if not excellent. About half of the 
participants felt their respective school faculties were ready and willing to implement school 
change. Another fourth felt that about half their faculties were ready at this point, but the 
remaining fourth felt implementation would be difficult at the present time. 

Team Reflection Logs. We conclude that, as the 1996 ITI Summer Institute progressed, 
participants moved from learning and gathering information to clarifying roles and collaborating 
to designing activities to reviewing and revising and, finally, to integrating the unit. As a result, 
their emotions ranged widely from curiosity, apprehension, and anticipation at the beginning to 
enthusiasm and understanding midway through to exhausted, relieved, and productive at the 
conclusion of the Institute. The Institute served as a positive modeling experience for 
participants, and they felt a sense of accomplishment upon mastering assessment rubrics. Further, 
participants felt the most beneficial activities of the Institute included multiple intelligences, 
rubrics, the mousetrap, and jigsaw. 

Overall Conclusions. We conclude, from the triangulation of the data garnered from the 
mixed-method approach, that the 1996 ITI Summer Institute had the greatest impact on 
participants in the following areas: collaboration, cooperative learning, assessment rubrics, 
multiple intelligences, and team building. Participants were overwhelmingly positive in their 
valuation of the Institute, which strengthened their belief in and commitment to ITI concepts and 
practices. Finally, after reviewing the outcomes of this comprehensive evaluation, we conclude 
that the quantitative instrumentation utilized may not have fully captured and documented 
institute effects. 
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RECOMMENDATIONS 



Several recommendations are offered, based upon findings and conclusions presented 

earlier. 



We recommend that the program continue offering ITI summer institutes to interested 
educators. Suggested improvements for future institutes might include more explanatory content, 
more time for individual activities, and slight scheduling adjustments. Also, we recommend that 
the program continue supporting curriculum integration efforts by keeping in touch with 
participants and providing new resources when available. One approach for this communication 
and information provision that is already being implemented by ITI staff is the use of their 
electronic ITI listserv, which connects interested educators from around the world. 

We recommend that the program continue evaluating such institutes, with both qualitative 
and quantitative methods. We do not feel the need, however, for such an intensive, compre- 
hensive evaluation effort as that presented in this report. The extensive evaluative methodology 
did not seem to be efficacious, in that the amount of information gained did not seem to be worth 
the prodigious amount of time spent gathering and analyzing data. Further, the utility of the three 
quantitative instruments used in this evaluation is questionable, since these surveys did not 
specifically assess the concepts and constructs undergirding the ITI activities and philosophies. 
We recommend that staff locate and secure other relevant instrumentation for future endeavors. 
And, staff might even consider alternate ways of focusing evaluative efforts to better document 
ITI’s ultimate impact on student outcomes. Staff might want to investigate in depth why some 
schools succeed in their ITI endeavors and report continuing implementation, while others 
apparently do not demonstrate similar success. One such approach might be site visits at selected 
ITI schools whose faculty have participated in previous summer institutes. 
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Sunday 

Theme : Introductions 



ITI Institute Agenda 



( Introductions to AEL, the Institute and facilitators . Getting 
acquainted with our own tearn members , ana meeting the other 
teams . ) 



3 : 00 


P . m . 


- 5:00 


p . m . 


Registration an 


d Check-in 


5:00 


p.m. 


- 6:00 


p.m. 


Dinner- -Dalton 


Hail 


6:00 


p.m. 


O • O 

U- . O 


p.m. 


Welcome 

Overview of Ins 


titute and 



ixpect&t 



Introductions 

Team Rc ies , Team Needs , Networking 
Journaling 

(Commonwealth Room, Heth Mall) 



Hall 



For the remainder of the week, all meals are in Dalton Hall and 
all working sessions are in Heth Hall. 



Monday 

Theme: Team Roles and Responsibilities 




7:30 a . m . 


“ 8:30 o . m . 


Administrator Breakfast 


6:30 a . m 


- 10:15 a . m . 


Developing Teams : KOJ.es, Kules, 

Relationships, Results 


10:15 a . m . 


~ 1C:3C a . m . 


Break 


10:30 a . m . 


— 12:00 a .m. 


Tools to Support Teamwork 
Concurrent. Sessions : 






Conflict Resolution (Cascades) 

Tools for Testing Consensus (Claytor) 
Group Roles and Functions (New River) 


12:00 p.m. 


- 1 p.m. 


Lunch 


1:00 p.m. 


- 4:00 p.m. 


Team Exploration and Debrief 


4:00 p.m. 


- 5:00 p.m. 


R Sc R 


5:00 p.m. 


0:00 p . rr. . 


Dinner 


.no 

0 . ut‘ ^/ . in . 


- 7:30 p.m 


Team Planning and Reflections 


7:30 p . m . 


8:30 p.m. 


Ice Cream Sociax 

c A BEST COPY AVAILABLE 

j0 



Tuesday 



Theme: Integration Models and Processes 

Designing and developing ps vsrrui and meaningful learning 
activities in support of an integrated un~t. 



7:30 a 


. n \ . 


- 3:30 a . m . 


Administrator Breakfast 


0:30 a 


. m. 


10:1 z* a . m . 


Five Stages of Integration 
Reviewing and Rating ITI Examples 


10:10 


a . rvi . 


- 10:30 a . m . 


Break 


- r - o n 

J. C. . sj t< 


ci . n: . 


- 1 _ : 0 G a . m • 


Lis covering "Fertile'' Themes 


12:00 


p . m . 


, - 1:00 p.rrs. 


Lunch 


1:00 P 


. m . 


- 2:00 p.m. 


Team. Applications: Unit resign 








Generating Learner Goals 
Identifying Powerful Learning 


2:00 p 


> .m. 


- 4:00 p .m . 


Team Planning Time: 

Beginning the Unit Development- - 
Selecting a Theme 
Framing Essential Questions 
Designing Powerful Learning Exper 



4:00 


p . m . 


- 5:00 p.m. 


R & Pv 


5:00 


p- . m. 


6 * 0 C p . TTi . 


Dinner 


6 : 00 


P • IT. . 


“ 7:3*0 p .ra. 


Evening Session 



V i d e o s 
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Wednesday 

Theme: Building Activities That Link with Assessment 



Administrator Breakfast 

Learning Principles and Authentic 

Concurrent sessions: 

Cooperative Learning (Claytor) 
Problem- Based Learning (New Rivs 
Multiple Intelligences (Cascades 



A well designed act 
activities with the 


ivity 

cuimi 


7:15 a.m. 


- 8 : 00 a. 


m . 


b : 0 0 rt . n*i . 


•• 9:15 a . 


TTi . 


9:15 a. it. - 10: 3C 


£; . m . 



BEST COPY AVAILABLE 



10:30 s.m. 


1 0 : 4 z> 3 . m • 


Break 


10:45 a . m . 


- 12:00 a.m. 


Concurrent. Sessions: 






Cooperative Tasks (Cl ay tor) 

Community -based Learning (New River) 
Writing to Learn (Cascades) 


12 : 0C p.m. 


1 : 0 0 p . rn . 


Lunch 


1:00 p . ui . 


- 2:00 p.m. 


Structuring a Powerful Learning 
Experience 


2: 00 p .m. 


- 4:00 p.m. 


Team Planning: Incorporating Authent. 

Tasks into Your Unit 






Team Reflections 


4:00 p . n'i . 


- 6:30 p . m . 


R & R 


6:30 p.m. 


- 8:30 p.m. 


Institute Picnic on the Lawn 


Thursday 






Theme: Focusing on Performance 


7:30 a . rti . 


- 6:30 a.m. 


Administrator Breakfast 


8:30 a.m. 


- 12:00 p.m. 


Understanding Alternative Assessment 


9:30 a . m . 


- 10:15 a . n . . 


Constructing Scoring Rubrics 


12:00 p.m, 


. - 1:00 P.m. 


Lunch 


1:00 p.m. 


- 4:00 p.m. 


Team Planning Time: 

Construct a Rubric to 
Assess Student Performance 
Construct a Rubric to Assess Unit 
Performance 
Team P.eflection 


4:00 p.m. 


- 5:00 p.m. 


R & R 


5:00 p.m. 


- 6:0 0 p . m . 


Dinner 


6:00 p . m . 


(as needed) 


Team Planning: Prepare Team Exhibit 
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BEST COPY AVAILABLE 



Friday 

Theme: Culminating Performances 



7:30 d.m. ~ 


S: 30 


a. m. 


Administrator Breakfast: 


8:30 a . m . - 


9:00 


a.m. 


Planning for Re-entry 


9:00 a.rri. - 


9: 30 


a.m. 


Team Action Plans 


9:30 a . m . * 


10 : 1 


5 a.m. 


Team Exhibitions 


10:15 a . m . 


- 10: 


30 a.m. 


Break 


10:30 a . m . 


* 1 • 


i : a . in . 


Team Exhibitions (continued) 


11:1: a . m . 


- ^ 9 


C 1 L-* p- • m * 


Pres^ntciiio:: or Certi: icdiri' 








Program ivai. Uation 
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Appendix B: 



Teacher Questionnaire 
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Teacher Questionnaire 



Last Four Social Security Numbers: Date: 



Directions: This questionnaire concerns your perceptions of how you view your teaching 
role. There are no right or wrong responses. Please read each numbered statement 
carefully. Then respond by circling one of the responses on the scale of Strongly Disagree 
(SD) to Strongly Agree (SA). Please do not skip any statements. 



Strongly 

Disagree 

1 


Disagree 

1 


Neither 
Disagree 
or Agree 


Agree 
| 


Strongly 
Agree 
1 


1 

SD 


1 

D 


I 

N 


1 

A 


i 

SA 



1. I am given the responsibility to monitor programs. 

SD D N A SA 

2. 1 function in a professional environment. 

SD D N A SA 

3. 1 believe that 1 have earned respect. 

SD D N A SA 

4. I believe that 1 am helping kids become independent 
learners. 

SD D N A SA 

5. I have control over daily schedules. 

SD D N A SA 

6. I believe that 1 have the ability to get things done. 

SD D N A SA 

7. I make decisions about the implementation of new 
programs in school. 

SD D N A SA 

8. I am treated as a professional. 

SD D N A SA 

9. I believe that I am very effective. 

SD D N A SA 



10. I believe that I am empowering students. 

SD D N A SA 

11. I am able to teach as I choose. 

SD D N A SA 

12. I participate in staff development. 

SD D N A SA 

13. 1 make decisions about the selection of other teachers for 
my school. 

SD D N A SA 

14. 1 have the opportunity for professional growth. 

SD D N A SA 

15. 1 have the respect of my colleagues. 

SD D N A SA 

16. 1 feel that 1 am involved in an important program 
for children. 

SD D N A SA 

17. 1 have the freedom to make decisions on what is taught. 

SD D N A SA 

18. 1 believe that 1 am having an impact. 

SD D N A SA 
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19. I am involved in school budget decisions. 

SD D N A SA 

20. 1 work at a school where kids come first. 

SD D N A SA 

21. I have the support and respect of my colleagues. 

SD D N A SA 

22. I see students leam. 

SD D N A SA 

23. 1 make decisions about curriculum. 

SD D N A SA 

24. 1 am a decision maker. 

SD D N A SA 

25. lam given the opportunity to collaborate with other 
teachers. 

SD D N A SA 

26. I believe that I have the ability to get things done. 

SD D N A SA 

27. I have a strong knowledge base in the areas in which I 
teach. 

SD D N A SA 

28. I believe that I have the ability to grow by working 
daily with students. 

SD D N A SA 

29. I perceive that I have the opportunity to influence 
others. 

SD D N A SA 

30. I can determine my own schedule. 

SD D N A SA 

31. I have the opportunity to collaborate with other teachers 
in my school. 

SD D N A SA 

32. I perceive that I am making a difference. 

SD D N A SA 



0 




33. Principals, other teachers, and school personnel solicit 
my advice. 

SD D N A SA 

34. I believe that 1 am good at what 1 do. 

SD D N A SA 

35. 1 can plan my own schedule. 

SD D N A SA 

36. 1 perceived that 1 am having an impact on other teachers 
and students. 

SD D N A SA 

37. My advice is solicited by others. 

SD D N A SA 

38. 1 have an opportunity to teach other teachers about in- 
novative ideas. 

SD D N A SA 

39. What grade (s) do you teach? 



40. What subject(s) do you 

teach? 

41. Counting this year, how many years have you taught ir 

any school? 

42. Do you teach full time or part time? (mark one) 

Full Time Part Time 

43. Counting this year, how many years have you taught in 

this school? 

44. Check the one category that describes how many de- 
grees and credits you have now. 

Bachelors Masters 

Bachelors +15 Masters +15 

Bachelors +30 Masters +30 

Bachelors +45 Masters +45 

Bachelors +60 Masters +60 

Education Specialists Degree 

Doctors Degree 

Other (Explain) 



45. Check one: Female Male 

46. Your age is years. 
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School as Community Questionnaire 
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School as Community Questionnaire 



Last Four Social Security Numbers: Date: 



Directions: This questionnaire concerns your perceptions of how much your school is a 
community. There are no right or wrong responses. Please read each numbered 
statement carefully. Then respond by circling one of the response options on the 
scale of Strongly Disagree (SD) to Strongly Agree (SA). Please respond in terms of this 



school, its staff, and students. 








Strongly 

Disagree 

i 


Disagree 

i 


Neither 
Disagree 
or Agree 
I 


Agree 

| 


Strongly 
Agree 
1 


1 

SD 


1 

D 


1 

N 


i 

A 


SA 



1. Most of my colleagues share my beliefs and values about 
what the central mission of the school should be. 

SD D N A SA 

2. Goals and priorities for the school are clear. 

SD D N A SA 

3. In this school, the teachers and the administration 
are in close agreement on school discipline policy. 

SD D N A SA 

4. Since the beginning of the current school year, I have 
received many useful suggestions for curriculum materi- 
als from colleagues in my department. 

SD D N A SA 

5. Since the beginning of the current school year, I have 
received many useful suggestions for teaching tech- 
niques or student activities from colleagues in my de- 
partment. 

SD D N A SA 

6. There is a great deal of cooperative effort among staff 
members. 

SD D N A SA 

7. I make a conscious effort to coordinate the content of my 
courses with other teachers. 

SD 

0 

ERIC 



8. In a typical planning period with other teachers, the 
group decides common themes and suggests related 
materials and activities to guide instruction. 

SD D N A SA 

9. Since the beginning of the school year, I have met with 
other teachers often regarding lesson planning, curricu- 
lum development, guidance and counseling, evaluation 
of programs, or other collaborative work related to in- 
struction. 

SD D N A SA 

10. Higher level skills (reasoning, problem solving, critical, 
and creative thinking) are important student goals of my 
teaching. 

SD D N A SA 

1 1. Teachers focus on what and how well students are 
learning rather than on how they are teaching. 

SD D N A SA 

12. Teachers exhibit a reasonably focused commitment to 
authentic curriculum and instruction. 

SD D N A SA 

13. A focused school vision for student learning is shared by 
most staff in the school. 

SD D N A SA 



D N A SA 
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14. Two or more teaching colleagues regularly observe my 
students' academic performance, or review their grades 
or test scores. 

SD D N A SA 

15. Except for monitoring student teacher or substitute 
teachers, I have often visited another teacher's classroom 
to observe and discuss their teaching since the begin- 
ning of the current school year. 

SD D N A SA 

16. Since the beginning of the current school year, I have 
received meaningful feedback on my performance from 
my supervisors or peers. 

SD D N A SA 

* 17. In a typical planning period with other teachers, the 
group discusses problems of specific students and ar- 
ranges appropriate help. 

SD D N A SA 

18. In a typical planning period with other teachers, the 
group discusses specific teaching practices and behav- 
iors of team members. 

SD D N A SA 

19. In formal and informal meetings of teachers, the group 
discusses the goals of the school. 

SD D N A SA 

20. In formal and informal meetings of teachers, the group 
discusses the teaching profession. 

SD D N A SA 



22. In formal and informal meetings of teachers, the group 
discusses the evaluation of student learning. 

SD D N A SA 

23. What grade(s) do you teach? 



24. What subject(s) do you teach? 



25. Counting this year, how many years have you taught in 

any school 

26. Do you teach full time or part time? (Mark one) 

Full Time Part Time 

27. Counting this year, how many years have you taught in 

this school? 

28. Check the one category that describes how many de- 
grees and credits you have now. 

Bachelors Masters 

Bachelors +15 Masters +15 

Bachelors +30 Masters +30 

Bachelors +45 Masters +45 

Bachelors +60 Masters +60 

Education Specialists Degree 

Doctors Degree 

Other (Explain) 



29. Check one: Female Male 

21. In formal and informal meetings of teachers, the group 

discusses how students learn. 30. Your age is years. 

SD D N A SA 
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School Products and Services Questionnaire 



School Products and Services Questionnaire 



Last Four Social Security Numbers: Date: 



Directions: These questions concern your perceptions of your school's overall effectiveness. There 
are no right or wrong answers. Every educator and staff member produces something during their 
work in the school. The following "products" and services are just a few of the things that result 
from schools' work: 

Lesson Plans Student Learning Co-Curricular Activities 

Curriculum Community Projects Parent Involvement 

Art & Music Programs Instruction Clean Schools 



Please indicate your response to each of the first eight numbered questions by circling a number 
from 1 (low) to 5 (high) on the scale provided under the question or by marking an "X" on scale. 



Of the various things produced by 
the people you know in your 
school, how much are they pro- 
ducing? 


Low 

Production 


Fairly Low 
Production 


Moderate 

Production 


High 

Production 

1 


Very High 
Production 


1 


2 


3 


1 

4 


5 


How good is the quality of the 
products or services produced by 
the people you know in your 
school? 


Poor 

Quality 


Low 

Quality 


Fair 

Quality 


Good 

Quality 


Excellent 

Quality 

I 


1 


2 


3 


4 


1 

5 


Do the people in your school get 
maximum output from the avail- 
able resource (money, people, 
equipment, etc.)? That is, how 
efficiently do they do their work? 


Not 

Efficiently 


Not Very 
Efficiently 


Fairly 

Efficiently 


Very 

Efficiently 


Extremely 

Efficiently 


i 


2 


3 


4 


5 


How good a job is done by the 
people in your school in 

antirnnatinrr r’trnhilpmQ anr! 


A Poor 
Job 
I 


An Adequate 
Job 


A Fair 
Job 


A Very Good 
Job 
1 


An Excellent 
Job 

1 


(XllUwiUQllilU LJ1 UUlCiUO CLllLl 

preventing them from occurring or 
minimizing their effects? 


1 

1 


2 


3 


i 

4 


1 

5 


How informed are the people in 
your school about innovations that 

rnnlH aff ort flip \A/av tLiPV Hn tVipir 


Uninformed 


Somewhat 

Informed 


Moderately 

Informed 


Informed 

I 


Very 

Informed 


OvJLLLU dlicvA Lllc vv ay nicy uu uicn 

work? 


1 


2 


3 


1 

4 


5 
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When changes are made in meth- 


Very 


Rather 


Fairly 






ods, routines, or equipment, how 
quickly do the people in your 


Slowly 


Slowly 


Rapidly 


Rapidly 


Immediately 


school accept and adjust to the 
changes? 


1 


2 


3 


4 


5 



How many of the people in your 


Few, if 


Less than 


About 


More Than 


Almost 


school readily accept and adjust to 
the changes? 


Any 


Half 

1 


Half 


Half 


Everyone 


1 


1 

2 


3 


4 


5 



8. How good a job do the people in 
your school do in coping with 
emergencies and disruptions? 



9. What is your role in the school? Principal/Asst. Principal Counselor/Psychiatrist 

(check only one) 

Classroom Teacher Teacher's Aide 

Itinerant Teacher (not at Bldg, full time) Title I Teacher 

School Staff (secretary, custodian, cook) Other (Explain: 



A Poor 


An Adequate 


A Fair 


A Very Good 


An Excellent 


Job 


Job 


Job 

1 


Job 

1 


Job 


1 


2 


1 

3 


1 

4 


5 



10. Do you teach full time or part 

time? (Mark one) Full Time Part Time 



11. Counting this year, how many 
years have you taught in this 

school? years 



12. Check the one category that de- Bachelors Masters 

scribes how many degrees and 
credits you have now. 

Bachelors +30 Masters +30 

Bachelors +45 Masters +45 

Bachelors +60 Masters +60 

Education Specialists Degree 

Doctors Degree 

Other (Explain) 



13. Check one: Female Male 



14. Your age is years. 

O 
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Appendix E: 
Control Questionnaire 



CONTROL QUESTIONNAIRE 



Date: 



For continuity in processing these data, please write in the last four digits of your Social Security 
number: 



Instructions: This is not a test--there are no right or wrong responses. Please read each num- 
bered statement carefully. Where there is a blank provided ( ), decide what your 

normal or usual attitude, feeling, or behavior would be from these choices: 



A 

Rarely 

(less than 10% 
of the time) 



B 

Occasionally 
(about 30% 
of the time) 



C 

Sometimes 
(about half 
of the time) 



D 

Frequently 

(about 70% 
of the time) 



U 

Usually 
(more than 
90% of the time) 



Of course, there are unsure situations in which this would not be the case, but think of what you 
would do or feel in most normal situations. 

Write the letter that describe your usual attitude or behavior in the blank space provided for each 
statement. Please do not skip any statements in the list. 



I 



1. When faced with a problem, I try 

to forget it. 

2. I need frequent encouragement 

from others for me to keep working at a 
difficult task. 

3. | like jobs where I can make 

decisions and be responsible for my own 
work. 

4. | change my opinion when some- 

one I admire disagrees with me. 

5. If I want something, I work hard to 

get it. 

6. I prefer to learn the facts about 

something from someone else rather than 
have to dig them out for myself. 

7. I will accept jobs that require me 

to supervise others. 

d ’ 




8. I have a hard time saying "no" 

when someone tries to sell me something I 
don't want. 

9. | like to have a say in any deci- 

sions made by any group I'm in. 

10. I consider the different sides of an 

issue before making any decisions. 

1 1 . What other people think has a 

great influence on my behavior. 

12. Whenever something good happens to me, I 
feel it is because I've earned it. 

13. | enjoy being in a position of 

leadership. 

14. I need someone else to praise 

my work before I am satisfied with what I've 
done. 




(Please turn this sheet over and continue) 






Instructions: This is not a test-there are no right or wrong responses. Please read each num- 
bered statement carefully. Where there is a blank provided ( ), decide what your 

normal or usual attitude, feeling, or behavior would be from these choices: 



A 

Rarely 

(less than 1 0% 
of the time) 



B 

Occasionally 
(about 30% 
of the time) 



C 

Sometimes 
(about half 
of the time) 



D 

Frequently 
(about 70% 
of the time) 



U 

Usually 
(more than 
90% of the time) 



Of course, there are unsure situations in which this would not be the case, but think of what you 
would do or feel in most normal situations. 

Write the letter that describe your usual attitude or behavior in the blank space provided for each 
statement. Please do not skip any statements in the list. 



•:V 

1 

1 






15. lam sure enough of my opinion 

to try and influence others. 

16. When something is going to affect me, I 
learn as much about it as I can do. 

17. I decide to do things on the spur 

of the moment. 

1 8. For me, knowing I’ve done something well is 

more important than being 

praised by someone else. 

19. I let other peoples’ demands keep 

me from doing things I want to do. 

20. I stick to my opinions when 

someone disagrees with me. 

21. I do what I feel like doing not 

what other people think I ought to do. 

22. I get discouraged when doing 

something that takes a long time to achieve 
results. 



23. When part of a group, I prefer to 

let other people make all the decisions. 

24. When I have a problem, I follow 

the advice of friends or relatives. 

25. I enjoy trying to do difficult tasks 

more than I enjoy trying to do easy tasks. 

26. I prefer situations where I can 

depend on someone else's ability rather than 
just my own. 

27. Having someone important tell me I did a 

good job is more important to me 

than feeling I've done a good job. 

28. When I'm involved in something, I 

try to find out all I can about what 

is going on even when someone else is in 
charge. 
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Feedback Form 
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Feedback on Dissolving the Boundaries Summer Institute 



Name: (optional) Date: 

Position: 

PLEASE RESPOND TO ALL OF THE ITEMS ON THIS FORM 



For the following items please circle the number that best indicates the extent to which this 
institute: 


Not at all 

1 . Had clear outcomes 1 


2 


-* 


4 


Very much 
5 


2. Promoted team building 1 






4 


5 


2'. Caused me to reflect on my practices 1 


2 


3 


4 


5 


4. Caused me to examine some of my attitudes 1 


'1 




4 


5 


5. Facilitated development of new skills 1 


o 




4 


> 


6. Was relevant to my needs 1 


o 


JV 


i 


■N 


7. Was conducted in a positive manner 1 


2 




4 


5 


8. Had activities that were well sequenced 1 


2 




4 


5 


9. Had activities that reinforced content 1 


2 


J) 


4 


N 


10. Included appropriate examples 1 


2 




1 


5 


1 1 . Was conducted by competent trainers 1 




■\ 




5 


12. Was conducted in a professional manner 1 


2 




4 


5 


13. Had meaningful involvement of participants 1 


2 




4 


5 


14. Has stimulated me to want to use the concepts 
skills, and materials in my position 1 


2 




4 


5 


15. Provided materials/ideas that will be useful 

to me in the future 1 


2 


3 


4 


5 


Please see other side of page for open-ended items and comments. 
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CONTINUED ^ 



16. 



How would you describe your institute experience to others? 



17. What new ideas, strategies, activities, resources were most meaningful for you? 



1 8. What do you consider the major strengths of the summer institute? 



19. What recommendations, if any. could you make for future institutes? 



O 

ERIC 
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20 . 



How has the institute changed the way you think about IT1? 



21 . How will you use what you have learned in your school next year? 



22. In what ways can AEL continue to support vour efforts with curriculum integration 



23. Other comments? 
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Appendix G: 
Interview Questions 



INTERDISCIPLINARY TEAMED INSTRUCTION (ITI) PROJECT 
Summer 1996 ITI Training Institute 
Participant Interview Questions 



Last four S. S. Digits: 



ITI Team Code: 



1 . How much decision-making authority do you have over what is taught and how it is taught? 



2. Describe ways in which you believe you are helping students become independent learners? 



3. How would your team members describe how well students are learning? 



4. How would you judge and explain your school team to be at collaborating on professional- 
type issues? 



5. How would you rate your school faculty on their readiness and willingness to accept 
changes? 
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Appendix H: 

Role- Alike Caucus Sheet 



Role-Alike Caucus Sheet 



Instructions: 

1. Assemble in your role-alike groups near the sign appropriate for your role group. 

2. Have everyone introduce themselves by name and school /districl/organizati on. 

3. Select a facilitator lo ask the caucus questions and keep the discussion moving, a timekeeper 
to remind the group of the time remaining to complete the activity, and a recorder to capture 
the kev ideas and phrases. Everybody should have an opportunity to speak. The questions 
are: 

A. What to you seems most promising by way of new approaches to teaching and learning? 

B. What special opportunities and challenges are associated with collaborating with others in 
designing interdisciplinary studies? 

ROLE-ALIKE GROUP: 

Names of participants: 



A. What to you seems most promising by way of new approaches to teaching and learning? 



B. What special opportunities and challenges are associated with collaborating with others in 
designing interdisciplinary studies? 



School Strategies & Options, P.O. Box 597, Lunenburg, MA 01462 (508) 582-4217 
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Appendix J: 



Presentation Scoring Rubric 



ITI INSTITUTE TEAM EXHIBITIONS 
A Holistic Rubric Based on Clustered Attributes 



Level 4 EXPERT - An exceptional performance. The story line and characters reflect a very 
high level of creativity, insight, or humor. The major messages developed at the week-long 
institute are clearly conveyed. 

Level 3 PRACTITIONER - The intended message reaches the audience. Everyone can be seen 
and heard. Props and related technologies enhance the message without intruding on or 
overshadowing it. 

Level 2 APPRENTICE - The message is mixed or missing, or participation is not balanced. 
Some elements may actually be done well, but this performance is distinguished by its 
unevenness. 

Level 1 NOVICE - Some of the team members cannot be heard or 

seen well There are gaps or pauses in the flow of the performance. Technologies or props may 
be lacking or even distracting. The whole program may appear not organized or is too 
long or too short. 
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Appendix K: 

Evaluation Standards Checklist 
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Citation Form 



I 

fH * 15 

I 



Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

evaluation report 

other: 



JLX 



I o interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
ommittee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



l 



Jescriptor 

U1 Stakeholder Identification 



The Standard was 
deemed applicable 
and to the extent 
feasible was taken 
into account. 

xxxx 



The Standard was 
deemed applicable 
but could not be 
taken into account. 



The Standard was 
not deemed appli- 
cable. 



Exception was taken 
to the Standard. 



I 



2 Evaluator Credibility 



XXXX 



3 Information Scope and Selection 



XXXX 



U4 Values Identification 



XXXX 



( 



5 Report Clarity 



XXXX 



6 Report Timeliness and Dissemination 



XXXXf Oatn 



analysis don 



e on time 



17 Evaluation Impact 



XXXX 



bn 



t report was 



not) 



FI Practical Procedures 



XXXX 



12 Political Viability 



XXXX 



F3 Cost Effectiveness 



XXXX 



PI Service Orientation 



XXXX 



I 

i 



2 Formal Agreements 



xxxx 



'3 Rights of Human Subjects 



XXXX 



'4 Human Interactions 



XXXX 



5 Complete and Fair Assessment 



XXXX 



6 Disclosure of Findings 



XXXX 



7 Conflict of Interest 



XXXX 



P8 Fiscal Responsibility 



XXXX 



ft 



Program Documentation 



XXXX 



Context Analysis 



XXXX 



A3 Described Purposes and Procedures 



XXXX 



t 

£ 

A1 

ft 



Defensible Information Sources 



XXXX 



Valid Information 



XXXX 



Reliable Information 



XXXX 



Systematic Information 



XXXX 



Analysis of Quantitative Information 



XXXX 



Analysis of Qualitative Information 



XXXX 



A10 Justified Conclusions 
1 Impart ial Reporting 
M2 Metaevaluation 



XXXX 



XXXX 



XXXX 



lame 



Merrill L, Meehan 



(typed) 



I 






Date: 12/98 



osition or Title: 



(signature) 

Senior R§E Specialist 



I Agency: Appalachia Educational Laboratory 

'Address: P-0. Box 1348, Charleston, WV 25325 
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ERIC (e.g., author of document, evaluation team leader, external auditor, internal auditor) 
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